Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 13:51:47.858116 416 test_util.cc:276] Using random seed: -1090225529
2025-06-25T13:51:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:51:47Z Disabled control of system clock
I20250625 13:51:47.892436 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33015
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 13:51:48.229032 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:51:48.229364 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:51:48.229449 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:51:48.237957 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:51:48.238066 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:51:48.238132 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:51:48.238180 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:51:48.247848 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:51:48.249436 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:51:48.251688 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:51:48.262161 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:48.262073 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:48.262101 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:48.262820 424 server_base.cc:1048] running on GCE node
I20250625 13:51:48.263823 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:51:48.265257 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:51:48.266485 424 hybrid_clock.cc:648] HybridClock initialized: now 1750859508266425 us; error 56 us; skew 500 ppm
I20250625 13:51:48.266868 424 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:51:48.269727 424 webserver.cc:469] Webserver started at http://127.0.104.62:44303/ using document root <none> and password file <none>
I20250625 13:51:48.270524 424 fs_manager.cc:362] Metadata directory not provided
I20250625 13:51:48.270644 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:51:48.271014 424 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:51:48.273602 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/instance:
uuid: "3fec9d8a79054a5991ae90d8a3101e32"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
I20250625 13:51:48.274394 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal/instance:
uuid: "3fec9d8a79054a5991ae90d8a3101e32"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
I20250625 13:51:48.279726 424 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250625 13:51:48.283067 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:48.284364 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:51:48.284529 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
uuid: "3fec9d8a79054a5991ae90d8a3101e32"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
I20250625 13:51:48.284718 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:51:48.302304 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:51:48.303138 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:51:48.303437 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:51:48.323860 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33015
I20250625 13:51:48.323868 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33015 every 8 connection(s)
I20250625 13:51:48.325311 424 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
I20250625 13:51:48.329396 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:48.329612 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 424
I20250625 13:51:48.330440 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal/instance
I20250625 13:51:48.333722 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:45727
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 13:51:48.338260 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
I20250625 13:51:48.356526 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } has no permanent_uuid. Determining permanent_uuid...
W20250625 13:51:48.357679 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45727: connect: Connection refused (error 111)
W20250625 13:51:48.359165 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45727: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45727: connect: Connection refused (error 111)
I20250625 13:51:48.408586 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } attempt: 1
W20250625 13:51:48.410522 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45727: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45727: connect: Connection refused (error 111)
I20250625 13:51:48.478792 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } attempt: 2
W20250625 13:51:48.480618 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45727: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45727: connect: Connection refused (error 111)
I20250625 13:51:48.571939 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } attempt: 3
W20250625 13:51:48.573747 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45727: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45727: connect: Connection refused (error 111)
W20250625 13:51:48.668260 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:51:48.668627 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:51:48.668756 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:51:48.677237 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:51:48.677361 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:51:48.677472 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:51:48.677539 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:51:48.687124 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:45727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:51:48.688791 492 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:51:48.690605 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:51:48.701258 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:48.701431 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:48.701377 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:48.702256 492 server_base.cc:1048] running on GCE node
I20250625 13:51:48.703002 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:51:48.704444 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:51:48.706096 492 hybrid_clock.cc:648] HybridClock initialized: now 1750859508705583 us; error 37 us; skew 500 ppm
I20250625 13:51:48.706451 492 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:51:48.712801 492 webserver.cc:469] Webserver started at http://127.0.104.61:38921/ using document root <none> and password file <none>
I20250625 13:51:48.713739 492 fs_manager.cc:362] Metadata directory not provided
I20250625 13:51:48.713845 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:51:48.714211 492 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:51:48.716960 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } attempt: 4
I20250625 13:51:48.717015 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data/instance:
uuid: "58351267767f4d9a85f974ebc40db6fe"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
I20250625 13:51:48.718024 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/wal/instance:
uuid: "58351267767f4d9a85f974ebc40db6fe"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
W20250625 13:51:48.718868 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45727: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45727: connect: Connection refused (error 111)
I20250625 13:51:48.723874 492 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250625 13:51:48.727108 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:48.728444 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 13:51:48.728605 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/wal
uuid: "58351267767f4d9a85f974ebc40db6fe"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
I20250625 13:51:48.728938 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:51:48.752815 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:51:48.753769 492 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:51:48.754045 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:51:48.775677 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:45727
I20250625 13:51:48.775700 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:45727 every 8 connection(s)
I20250625 13:51:48.777128 492 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/data/info.pb
I20250625 13:51:48.780927 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:48.782873 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 492
I20250625 13:51:48.783066 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-1/wal/instance
I20250625 13:51:48.784391 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43307
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 13:51:48.789525 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
I20250625 13:51:48.804119 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } has no permanent_uuid. Determining permanent_uuid...
I20250625 13:51:48.813484 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
W20250625 13:51:48.814702 513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
W20250625 13:51:48.816272 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:51:48.865583 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 1
W20250625 13:51:48.867393 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:51:48.935653 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 2
W20250625 13:51:48.937470 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:51:49.018111 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } attempt: 5
I20250625 13:51:49.025375 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
W20250625 13:51:49.027096 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:51:49.028710 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 3
W20250625 13:51:49.030439 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:51:49.078400 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 1
W20250625 13:51:49.080227 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
W20250625 13:51:49.122328 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:51:49.122670 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:51:49.122810 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:51:49.130939 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:51:49.131045 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:51:49.131148 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:51:49.131213 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:51:49.141191 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:51:49.142901 563 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:51:49.144702 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 13:51:49.148579 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 2
W20250625 13:51:49.150653 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
W20250625 13:51:49.155421 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:49.155469 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:49.155771 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:49.156720 563 server_base.cc:1048] running on GCE node
I20250625 13:51:49.157523 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:51:49.158972 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:51:49.160161 563 hybrid_clock.cc:648] HybridClock initialized: now 1750859509160112 us; error 42 us; skew 500 ppm
I20250625 13:51:49.160502 563 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:51:49.162734 563 webserver.cc:469] Webserver started at http://127.0.104.60:37999/ using document root <none> and password file <none>
I20250625 13:51:49.163542 563 fs_manager.cc:362] Metadata directory not provided
I20250625 13:51:49.163650 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:51:49.164004 563 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:51:49.166327 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data/instance:
uuid: "d85834f366fa42c294c9290af21ea111"
format_stamp: "Formatted at 2025-06-25 13:51:49 on dist-test-slave-x7jc"
I20250625 13:51:49.167078 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal/instance:
uuid: "d85834f366fa42c294c9290af21ea111"
format_stamp: "Formatted at 2025-06-25 13:51:49 on dist-test-slave-x7jc"
I20250625 13:51:49.172256 563 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250625 13:51:49.173646 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 4
W20250625 13:51:49.175364 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:43307: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:51:49.175784 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:49.177079 563 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250625 13:51:49.177210 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal
uuid: "d85834f366fa42c294c9290af21ea111"
format_stamp: "Formatted at 2025-06-25 13:51:49 on dist-test-slave-x7jc"
I20250625 13:51:49.177373 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:51:49.192952 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:51:49.193830 563 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:51:49.194063 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:51:49.214956 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:43307
I20250625 13:51:49.214963 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:43307 every 8 connection(s)
I20250625 13:51:49.216384 563 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data/info.pb
I20250625 13:51:49.220160 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:49.223682 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 563
I20250625 13:51:49.223891 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal/instance
I20250625 13:51:49.227020 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
I20250625 13:51:49.242045 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } has no permanent_uuid. Determining permanent_uuid...
I20250625 13:51:49.249801 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } has no permanent_uuid. Determining permanent_uuid...
I20250625 13:51:49.256886 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 3
I20250625 13:51:49.264115 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Bootstrap starting.
I20250625 13:51:49.268153 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:49.268555 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap starting.
I20250625 13:51:49.270092 632 log.cc:826] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Log is configured to *not* fsync() on all Append() calls
I20250625 13:51:49.272629 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:49.273221 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: No bootstrap required, opened a new log
I20250625 13:51:49.274012 490 log.cc:826] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Log is configured to *not* fsync() on all Append() calls
I20250625 13:51:49.276892 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: No bootstrap required, opened a new log
I20250625 13:51:49.280086 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.280439 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:49.280539 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85834f366fa42c294c9290af21ea111, State: Initialized, Role: FOLLOWER
I20250625 13:51:49.281350 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.282356 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.282709 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:49.283632 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:51:49.284300 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.284615 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:49.284722 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fec9d8a79054a5991ae90d8a3101e32, State: Initialized, Role: FOLLOWER
I20250625 13:51:49.285480 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.286379 648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.286603 648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:49.287582 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: configured and running, proceeding with master startup.
W20250625 13:51:49.295540 656 catalog_manager.cc:1560] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 13:51:49.295753 656 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 13:51:49.293320 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:33015: OK, instance_id { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" instance_seqno: 1750859508318661 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 13:51:49.299403 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 13:51:49.306027 444 tablet.cc:2378] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 13:51:49.310124 666 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 13:51:49.310273 666 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 13:51:49.307289 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:45727: OK, instance_id { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" instance_seqno: 1750859508770058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 13:51:49.314846 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 13:51:49.410073 642 raft_consensus.cc:491] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:49.410398 642 raft_consensus.cc:513] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.412575 642 leader_election.cc:290] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015), 58351267767f4d9a85f974ebc40db6fe (127.0.104.61:45727)
I20250625 13:51:49.414048 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d85834f366fa42c294c9290af21ea111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fec9d8a79054a5991ae90d8a3101e32" is_pre_election: true
I20250625 13:51:49.414590 465 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85834f366fa42c294c9290af21ea111 in term 0.
I20250625 13:51:49.415262 581 leader_election.cc:304] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fec9d8a79054a5991ae90d8a3101e32, d85834f366fa42c294c9290af21ea111; no voters:
I20250625 13:51:49.415716 642 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:49.415903 642 raft_consensus.cc:491] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:49.416009 642 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:49.419274 642 raft_consensus.cc:513] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.419996 642 leader_election.cc:290] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [CANDIDATE]: Term 1 election: Requested vote from peers 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015), 58351267767f4d9a85f974ebc40db6fe (127.0.104.61:45727)
I20250625 13:51:49.420727 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d85834f366fa42c294c9290af21ea111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fec9d8a79054a5991ae90d8a3101e32"
I20250625 13:51:49.420931 465 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:49.424319 465 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85834f366fa42c294c9290af21ea111 in term 1.
I20250625 13:51:49.424855 581 leader_election.cc:304] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fec9d8a79054a5991ae90d8a3101e32, d85834f366fa42c294c9290af21ea111; no voters:
I20250625 13:51:49.425282 642 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:49.425959 642 raft_consensus.cc:695] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [term 1 LEADER]: Becoming Leader. State: Replica: d85834f366fa42c294c9290af21ea111, State: Running, Role: LEADER
I20250625 13:51:49.427029 642 consensus_queue.cc:237] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.433145 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85834f366fa42c294c9290af21ea111. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.433537 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: LEADER
I20250625 13:51:49.441843 670 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:51:49.444577 670 catalog_manager.cc:1486] Initializing Kudu cluster ID...
W20250625 13:51:49.446017 586 tablet.cc:2378] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:51:49.457172 465 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Refusing update from remote peer d85834f366fa42c294c9290af21ea111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:51:49.458101 668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:49.474576 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } attempt: 5
I20250625 13:51:49.479341 648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85834f366fa42c294c9290af21ea111. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.479611 648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:49.487062 672 mvcc.cc:204] Tried to move back new op lower bound from 7171520550718967808 to 7171520550619144192. Current Snapshot: MvccSnapshot[applied={T|T < 7171520550718967808}]
I20250625 13:51:49.487242 671 mvcc.cc:204] Tried to move back new op lower bound from 7171520550718967808 to 7171520550619144192. Current Snapshot: MvccSnapshot[applied={T|T < 7171520550718967808}]
I20250625 13:51:49.488498 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe: Bootstrap starting.
I20250625 13:51:49.491942 648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.492154 648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:49.493189 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:49.500298 561 log.cc:826] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe: Log is configured to *not* fsync() on all Append() calls
I20250625 13:51:49.502110 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.502737 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: LEADER
I20250625 13:51:49.503589 670 catalog_manager.cc:1349] Generated new cluster ID: 4fb72e0fa75741a1972554db6ec14e0e
I20250625 13:51:49.503724 670 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:51:49.504786 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe: No bootstrap required, opened a new log
I20250625 13:51:49.512048 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.512530 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:49.512698 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58351267767f4d9a85f974ebc40db6fe, State: Initialized, Role: FOLLOWER
I20250625 13:51:49.513638 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:49.520753 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.522826 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:49.522704 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:51:49.522773 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d85834f366fa42c294c9290af21ea111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58351267767f4d9a85f974ebc40db6fe"
I20250625 13:51:49.523485 534 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:49.527123 534 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 1 FOLLOWER]: Refusing update from remote peer d85834f366fa42c294c9290af21ea111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:51:49.527621 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d85834f366fa42c294c9290af21ea111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58351267767f4d9a85f974ebc40db6fe" is_pre_election: true
I20250625 13:51:49.528473 536 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85834f366fa42c294c9290af21ea111 in term 1.
I20250625 13:51:49.528746 642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:49.539196 670 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:51:49.541265 670 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:51:49.565438 670 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Generated new TSK 0
I20250625 13:51:49.566810 670 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:51:49.572209 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85834f366fa42c294c9290af21ea111. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.572494 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:49.573249 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.573455 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: LEADER
I20250625 13:51:49.573891 683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:49.574199 683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:49.584196 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 13:51:49.652902 695 catalog_manager.cc:1261] Loaded cluster ID: 4fb72e0fa75741a1972554db6ec14e0e
I20250625 13:51:49.653100 695 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe: loading cluster ID for follower catalog manager: success
I20250625 13:51:49.661641 695 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe: acquiring CA information for follower catalog manager: success
I20250625 13:51:49.664322 695 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250625 13:51:50.031862 692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:51:50.032213 692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:51:50.032342 692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:51:50.040977 692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:51:50.041199 692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250625 13:51:50.051002 692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:51:50.052657 692 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:51:50.054401 692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:51:50.066256 702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:50.066540 704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:50.066509 701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:50.067196 692 server_base.cc:1048] running on GCE node
I20250625 13:51:50.067780 692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:51:50.068665 692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:51:50.069870 692 hybrid_clock.cc:648] HybridClock initialized: now 1750859510069808 us; error 61 us; skew 500 ppm
I20250625 13:51:50.070171 692 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:51:50.073359 692 webserver.cc:469] Webserver started at http://127.0.104.1:45175/ using document root <none> and password file <none>
I20250625 13:51:50.074179 692 fs_manager.cc:362] Metadata directory not provided
I20250625 13:51:50.074296 692 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:51:50.074745 692 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:51:50.077409 692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data/instance:
uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
format_stamp: "Formatted at 2025-06-25 13:51:50 on dist-test-slave-x7jc"
I20250625 13:51:50.078193 692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/instance:
uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
format_stamp: "Formatted at 2025-06-25 13:51:50 on dist-test-slave-x7jc"
I20250625 13:51:50.083698 692 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250625 13:51:50.087080 710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:50.088464 692 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250625 13:51:50.088636 692 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal
uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
format_stamp: "Formatted at 2025-06-25 13:51:50 on dist-test-slave-x7jc"
I20250625 13:51:50.088865 692 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:51:50.110850 692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:51:50.111804 692 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:51:50.112058 692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:51:50.113157 692 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:51:50.114924 692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:51:50.115021 692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:50.115123 692 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:51:50.115191 692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:50.155556 692 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:44801
I20250625 13:51:50.155608 822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:44801 every 8 connection(s)
I20250625 13:51:50.157119 692 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/data/info.pb
I20250625 13:51:50.164224 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 692
I20250625 13:51:50.164422 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/instance
I20250625 13:51:50.170291 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 13:51:50.186623 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:50.187122 823 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:50.187287 825 heartbeater.cc:344] Connected to a master server at 127.0.104.61:45727
I20250625 13:51:50.187443 825 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:50.188346 825 heartbeater.cc:507] Master 127.0.104.61:45727 requested a full tablet report, sending...
I20250625 13:51:50.188369 823 heartbeater.cc:507] Master 127.0.104.62:33015 requested a full tablet report, sending...
I20250625 13:51:50.190953 526 ts_manager.cc:194] Registered new tserver with Master: 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:50.191031 455 ts_manager.cc:194] Registered new tserver with Master: 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:50.193264 824 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43307
I20250625 13:51:50.193403 824 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:50.193915 824 heartbeater.cc:507] Master 127.0.104.60:43307 requested a full tablet report, sending...
I20250625 13:51:50.195374 597 ts_manager.cc:194] Registered new tserver with Master: 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:50.196862 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:53157
I20250625 13:51:50.312252 666 catalog_manager.cc:1261] Loaded cluster ID: 4fb72e0fa75741a1972554db6ec14e0e
I20250625 13:51:50.312358 666 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: loading cluster ID for follower catalog manager: success
I20250625 13:51:50.314944 666 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: acquiring CA information for follower catalog manager: success
I20250625 13:51:50.316776 666 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250625 13:51:50.508472 833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:51:50.508786 833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:51:50.508919 833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:51:50.517382 833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:51:50.517594 833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250625 13:51:50.527206 833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:51:50.528889 833 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:51:50.530640 833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:51:50.542388 838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:50.542685 841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:50.542408 839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:50.543170 833 server_base.cc:1048] running on GCE node
I20250625 13:51:50.543809 833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:51:50.544668 833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:51:50.545785 844 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d85834f366fa42c294c9290af21ea111)
I20250625 13:51:50.545850 833 hybrid_clock.cc:648] HybridClock initialized: now 1750859510545816 us; error 26 us; skew 500 ppm
I20250625 13:51:50.545957 844 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:50.546181 833 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:51:50.547704 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58351267767f4d9a85f974ebc40db6fe" is_pre_election: true
I20250625 13:51:50.547756 844 leader_election.cc:290] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58351267767f4d9a85f974ebc40db6fe (127.0.104.61:45727), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
I20250625 13:51:50.548403 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d85834f366fa42c294c9290af21ea111" is_pre_election: true
I20250625 13:51:50.548638 833 webserver.cc:469] Webserver started at http://127.0.104.2:44423/ using document root <none> and password file <none>
I20250625 13:51:50.549191 442 leader_election.cc:304] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fec9d8a79054a5991ae90d8a3101e32; no voters: 58351267767f4d9a85f974ebc40db6fe, d85834f366fa42c294c9290af21ea111
I20250625 13:51:50.549513 833 fs_manager.cc:362] Metadata directory not provided
I20250625 13:51:50.549562 844 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:51:50.549643 833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:51:50.550000 833 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:51:50.553000 833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data/instance:
uuid: "affa1c3035b940e88ca143be7d35840d"
format_stamp: "Formatted at 2025-06-25 13:51:50 on dist-test-slave-x7jc"
I20250625 13:51:50.553848 833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/instance:
uuid: "affa1c3035b940e88ca143be7d35840d"
format_stamp: "Formatted at 2025-06-25 13:51:50 on dist-test-slave-x7jc"
I20250625 13:51:50.558938 833 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250625 13:51:50.562111 848 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:50.563441 833 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:51:50.563591 833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal
uuid: "affa1c3035b940e88ca143be7d35840d"
format_stamp: "Formatted at 2025-06-25 13:51:50 on dist-test-slave-x7jc"
I20250625 13:51:50.563792 833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:51:50.576558 833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:51:50.577342 833 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:51:50.577575 833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:51:50.578411 833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:51:50.580032 833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:51:50.580148 833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:50.580260 833 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:51:50.580329 833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:50.616174 833 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:34247
I20250625 13:51:50.616222 960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:34247 every 8 connection(s)
I20250625 13:51:50.617589 833 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/data/info.pb
I20250625 13:51:50.625102 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 833
I20250625 13:51:50.625346 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/instance
I20250625 13:51:50.634315 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 13:51:50.643244 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:45727
I20250625 13:51:50.643757 963 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:50.644990 963 heartbeater.cc:507] Master 127.0.104.61:45727 requested a full tablet report, sending...
I20250625 13:51:50.645926 962 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43307
I20250625 13:51:50.646126 962 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:50.646610 526 ts_manager.cc:194] Registered new tserver with Master: affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:50.646674 962 heartbeater.cc:507] Master 127.0.104.60:43307 requested a full tablet report, sending...
I20250625 13:51:50.647601 597 ts_manager.cc:194] Registered new tserver with Master: affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:50.648699 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:52963
I20250625 13:51:50.649910 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:50.650035 961 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:50.650417 961 heartbeater.cc:507] Master 127.0.104.62:33015 requested a full tablet report, sending...
I20250625 13:51:50.651368 455 ts_manager.cc:194] Registered new tserver with Master: affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:50.699225 824 heartbeater.cc:499] Master 127.0.104.60:43307 was elected leader, sending a full tablet report...
W20250625 13:51:50.981006 968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:51:50.981307 968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:51:50.981397 968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:51:50.989055 968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:51:50.989257 968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250625 13:51:50.999217 968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:51:51.000846 968 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:51:51.002578 968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:51:51.015334 974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:51.015414 973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:51.015370 976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:51.019204 968 server_base.cc:1048] running on GCE node
I20250625 13:51:51.019850 968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:51:51.020764 968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:51:51.021960 968 hybrid_clock.cc:648] HybridClock initialized: now 1750859511021915 us; error 39 us; skew 500 ppm
I20250625 13:51:51.022248 968 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:51:51.024482 968 webserver.cc:469] Webserver started at http://127.0.104.3:35653/ using document root <none> and password file <none>
I20250625 13:51:51.025216 968 fs_manager.cc:362] Metadata directory not provided
I20250625 13:51:51.025326 968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:51:51.025679 968 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:51:51.028081 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data/instance:
uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
format_stamp: "Formatted at 2025-06-25 13:51:51 on dist-test-slave-x7jc"
I20250625 13:51:51.028825 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/instance:
uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
format_stamp: "Formatted at 2025-06-25 13:51:51 on dist-test-slave-x7jc"
I20250625 13:51:51.035037 968 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250625 13:51:51.038419 982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:51.039820 968 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 13:51:51.039984 968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal
uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
format_stamp: "Formatted at 2025-06-25 13:51:51 on dist-test-slave-x7jc"
I20250625 13:51:51.040179 968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:51:51.068884 968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:51:51.069783 968 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:51:51.070068 968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:51:51.071029 968 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:51:51.072711 968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:51:51.072814 968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:51.072930 968 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:51:51.073009 968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:51.108901 968 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:39291
I20250625 13:51:51.108969 1094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:39291 every 8 connection(s)
I20250625 13:51:51.110244 968 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/data/info.pb
I20250625 13:51:51.111119 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 968
I20250625 13:51:51.111346 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/instance
I20250625 13:51:51.136945 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43307
I20250625 13:51:51.137053 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:51.136965 1097 heartbeater.cc:344] Connected to a master server at 127.0.104.61:45727
I20250625 13:51:51.137490 1095 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:51.137490 1096 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:51.138252 1097 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:51.138715 1095 heartbeater.cc:507] Master 127.0.104.62:33015 requested a full tablet report, sending...
I20250625 13:51:51.138845 1097 heartbeater.cc:507] Master 127.0.104.61:45727 requested a full tablet report, sending...
I20250625 13:51:51.139816 1096 heartbeater.cc:507] Master 127.0.104.60:43307 requested a full tablet report, sending...
I20250625 13:51:51.140354 454 ts_manager.cc:194] Registered new tserver with Master: 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:51.140657 597 ts_manager.cc:194] Registered new tserver with Master: 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:51.141561 526 ts_manager.cc:194] Registered new tserver with Master: 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:51.141749 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:55967
I20250625 13:51:51.151194 962 heartbeater.cc:499] Master 127.0.104.60:43307 was elected leader, sending a full tablet report...
I20250625 13:51:51.158084 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:51:51.188445 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_29dd121f4cd84bf7b668fd8fcb03d4e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.189041 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f009bfdb9e314295855c61b4a1ab556f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.190384 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29dd121f4cd84bf7b668fd8fcb03d4e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:51.190418 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f009bfdb9e314295855c61b4a1ab556f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.190711 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_31842d3cc41c4282aa54922f28b7a212"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.191478 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31842d3cc41c4282aa54922f28b7a212 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.188375 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c353710775d94d2f8ce363dc94e1c586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.192507 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c353710775d94d2f8ce363dc94e1c586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.193017 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d837899f74444b0a8e38179bb8dea2ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.193840 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a13af807e80443588d6008d9ce3d727c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.196149 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d651af8c19894c9a96498b921d2fa6ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.196779 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d651af8c19894c9a96498b921d2fa6ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.198000 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.198997 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:51.199707 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a13af807e80443588d6008d9ce3d727c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:51.202572 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d837899f74444b0a8e38179bb8dea2ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.204407 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_17ac1814632146e081ee735425d7b2ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.204550 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d64bbacfddeb4945b2396e41e59cbb23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.204995 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17ac1814632146e081ee735425d7b2ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:51.205401 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d64bbacfddeb4945b2396e41e59cbb23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.281805 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu pid 424
I20250625 13:51:51.283211 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 13:51:51.287829 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_29dd121f4cd84bf7b668fd8fcb03d4e2" } new_table_name: "default.Table_755b00bf52dc402aaa1893a86293d177" modify_external_catalogs: true
I20250625 13:51:51.288698 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_31842d3cc41c4282aa54922f28b7a212" } new_table_name: "default.Table_f0748f143cf540dbbe1d769b411ac975" modify_external_catalogs: true
I20250625 13:51:51.289587 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd" } modify_external_catalogs: true
I20250625 13:51:51.289974 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d837899f74444b0a8e38179bb8dea2ee" } modify_external_catalogs: true
I20250625 13:51:51.290122 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f009bfdb9e314295855c61b4a1ab556f" } new_table_name: "default.Table_95be83c153474705b223639500ba9ff3" modify_external_catalogs: true
I20250625 13:51:51.290458 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d64bbacfddeb4945b2396e41e59cbb23" } modify_external_catalogs: true
I20250625 13:51:51.290848 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c353710775d94d2f8ce363dc94e1c586" } new_table_name: "default.Table_20b0b8fa5c384bbcb0fb51c0ecfd6a59" modify_external_catalogs: true
I20250625 13:51:51.291218 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d651af8c19894c9a96498b921d2fa6ca" } new_table_name: "default.Table_7533c0ca831e480380ee740fa1ec802c" modify_external_catalogs: true
I20250625 13:51:51.292110 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a13af807e80443588d6008d9ce3d727c" } modify_external_catalogs: true
I20250625 13:51:51.292300 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a13af807e80443588d6008d9ce3d727c" } modify_external_catalogs: true
I20250625 13:51:51.292855 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d64bbacfddeb4945b2396e41e59cbb23" } modify_external_catalogs: true
I20250625 13:51:51.294129 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d837899f74444b0a8e38179bb8dea2ee" } modify_external_catalogs: true
I20250625 13:51:51.296022 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd" } modify_external_catalogs: true
I20250625 13:51:51.296717 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_17ac1814632146e081ee735425d7b2ee" } modify_external_catalogs: true
I20250625 13:51:51.296906 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_17ac1814632146e081ee735425d7b2ee" } modify_external_catalogs: true
W20250625 13:51:51.300724 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab41f8d02dff471ba272dbf7620b9c6f has no consensus state
W20250625 13:51:51.304570 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5afade7a5d1447a98934e4d614075d8 has no consensus state
W20250625 13:51:51.305316 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d92711284e64f71b31666800c250eab consensus state has no leader
W20250625 13:51:51.306114 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d455a2dec4304cf48ce29ceeb0700b0f consensus state has no leader
W20250625 13:51:51.315477 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ecd6be619b79429b956aae4a1bf1589d consensus state has no leader
I20250625 13:51:51.322572 588 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a15a4979e3e9441c9a60c3a423344809
I20250625 13:51:51.326575 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 969dc4a9b811496aa3d926a3ae3965ea
I20250625 13:51:51.327662 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c87280e6a3e84e6ba5b0a90e1e7ecec3
I20250625 13:51:51.328599 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 20026a988f644f559ef967bcf9b46ed7
I20250625 13:51:51.330528 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 07458e047b88493e9e52423b4e85b091
W20250625 13:51:51.332639 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ab41f8d02dff471ba272dbf7620b9c6f (table default.Table_755b00bf52dc402aaa1893a86293d177 [id=d47dd54b4d984cc1874992118033256d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ab41f8d02dff471ba272dbf7620b9c6f consensus state has no leader
I20250625 13:51:51.349448 756 tablet_service.cc:1468] Processing CreateTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea (DEFAULT_TABLE table=default.Table_d64bbacfddeb4945b2396e41e59cbb23 [id=f00b1eb97d6245a8b22ad21216ed5230]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.351877 754 tablet_service.cc:1468] Processing CreateTablet for tablet a5afade7a5d1447a98934e4d614075d8 (DEFAULT_TABLE table=default.Table_95be83c153474705b223639500ba9ff3 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.350528 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.352612 752 tablet_service.cc:1468] Processing CreateTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f (DEFAULT_TABLE table=default.Table_20b0b8fa5c384bbcb0fb51c0ecfd6a59 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.349860 751 tablet_service.cc:1468] Processing CreateTablet for tablet 20026a988f644f559ef967bcf9b46ed7 (DEFAULT_TABLE table=default.Table_a13af807e80443588d6008d9ce3d727c [id=e88514d0a01b411ba21f2fd2b4d615e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.353466 750 tablet_service.cc:1468] Processing CreateTablet for tablet 07458e047b88493e9e52423b4e85b091 (DEFAULT_TABLE table=default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd [id=aa9bb127a9054fe18cee9dded1f88cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.349480 755 tablet_service.cc:1468] Processing CreateTablet for tablet 6d92711284e64f71b31666800c250eab (DEFAULT_TABLE table=default.Table_7533c0ca831e480380ee740fa1ec802c [id=96be558f5b694f3483d2501659dc5097]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.353873 748 tablet_service.cc:1515] Processing DeleteTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.353837 753 tablet_service.cc:1468] Processing CreateTablet for tablet ecd6be619b79429b956aae4a1bf1589d (DEFAULT_TABLE table=default.Table_f0748f143cf540dbbe1d769b411ac975 [id=d298fec0d0c14fc297e4a3b4f82b007a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.354238 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 07458e047b88493e9e52423b4e85b091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.354277 757 tablet_service.cc:1468] Processing CreateTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f (DEFAULT_TABLE table=default.Table_755b00bf52dc402aaa1893a86293d177 [id=d47dd54b4d984cc1874992118033256d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.354786 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 20026a988f644f559ef967bcf9b46ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.355482 758 tablet_service.cc:1468] Processing CreateTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 (DEFAULT_TABLE table=default.Table_d837899f74444b0a8e38179bb8dea2ee [id=bda1dd723374447b8a351345d7d3ae43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.357172 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969dc4a9b811496aa3d926a3ae3965ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.358183 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07458e047b88493e9e52423b4e85b091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.358472 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20026a988f644f559ef967bcf9b46ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.361559 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5afade7a5d1447a98934e4d614075d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.363128 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d92711284e64f71b31666800c250eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.364395 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d455a2dec4304cf48ce29ceeb0700b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.365512 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd6be619b79429b956aae4a1bf1589d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.365979 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab41f8d02dff471ba272dbf7620b9c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.366925 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:51.377727 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 969dc4a9b811496aa3d926a3ae3965ea because the tablet was not found. No further retry: Not found: Tablet not found: 969dc4a9b811496aa3d926a3ae3965ea
W20250625 13:51:51.378299 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 because the tablet was not found. No further retry: Not found: Tablet not found: c87280e6a3e84e6ba5b0a90e1e7ecec3
W20250625 13:51:51.378512 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 07458e047b88493e9e52423b4e85b091 because the tablet was not found. No further retry: Not found: Tablet not found: 07458e047b88493e9e52423b4e85b091
W20250625 13:51:51.378667 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 20026a988f644f559ef967bcf9b46ed7 because the tablet was not found. No further retry: Not found: Tablet not found: 20026a988f644f559ef967bcf9b46ed7
I20250625 13:51:51.379696 1166 tablet_bootstrap.cc:492] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:51.389784 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.395169 896 tablet_service.cc:1468] Processing CreateTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 (DEFAULT_TABLE table=default.Table_d837899f74444b0a8e38179bb8dea2ee [id=bda1dd723374447b8a351345d7d3ae43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.393347 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 20026a988f644f559ef967bcf9b46ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.396765 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.392210 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.398067 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 07458e047b88493e9e52423b4e85b091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.400017 895 tablet_service.cc:1468] Processing CreateTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f (DEFAULT_TABLE table=default.Table_755b00bf52dc402aaa1893a86293d177 [id=d47dd54b4d984cc1874992118033256d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.399574 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 20026a988f644f559ef967bcf9b46ed7 (DEFAULT_TABLE table=default.Table_a13af807e80443588d6008d9ce3d727c [id=e88514d0a01b411ba21f2fd2b4d615e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.400086 894 tablet_service.cc:1468] Processing CreateTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea (DEFAULT_TABLE table=default.Table_d64bbacfddeb4945b2396e41e59cbb23 [id=f00b1eb97d6245a8b22ad21216ed5230]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.400625 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab41f8d02dff471ba272dbf7620b9c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.400959 892 tablet_service.cc:1468] Processing CreateTablet for tablet a5afade7a5d1447a98934e4d614075d8 (DEFAULT_TABLE table=default.Table_95be83c153474705b223639500ba9ff3 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.401119 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20026a988f644f559ef967bcf9b46ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.402014 1030 tablet_service.cc:1468] Processing CreateTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 (DEFAULT_TABLE table=default.Table_d837899f74444b0a8e38179bb8dea2ee [id=bda1dd723374447b8a351345d7d3ae43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.403040 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.403540 1025 tablet_service.cc:1468] Processing CreateTablet for tablet ecd6be619b79429b956aae4a1bf1589d (DEFAULT_TABLE table=default.Table_f0748f143cf540dbbe1d769b411ac975 [id=d298fec0d0c14fc297e4a3b4f82b007a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.404074 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd6be619b79429b956aae4a1bf1589d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.405550 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6d92711284e64f71b31666800c250eab (DEFAULT_TABLE table=default.Table_7533c0ca831e480380ee740fa1ec802c [id=96be558f5b694f3483d2501659dc5097]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.405537 893 tablet_service.cc:1468] Processing CreateTablet for tablet 6d92711284e64f71b31666800c250eab (DEFAULT_TABLE table=default.Table_7533c0ca831e480380ee740fa1ec802c [id=96be558f5b694f3483d2501659dc5097]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.405979 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d92711284e64f71b31666800c250eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.406064 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 07458e047b88493e9e52423b4e85b091 (DEFAULT_TABLE table=default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd [id=aa9bb127a9054fe18cee9dded1f88cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.406601 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07458e047b88493e9e52423b4e85b091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.410247 1026 tablet_service.cc:1468] Processing CreateTablet for tablet a5afade7a5d1447a98934e4d614075d8 (DEFAULT_TABLE table=default.Table_95be83c153474705b223639500ba9ff3 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.410847 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5afade7a5d1447a98934e4d614075d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:51.412288 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 because the tablet was not found. No further retry: Not found: Tablet not found: c87280e6a3e84e6ba5b0a90e1e7ecec3
W20250625 13:51:51.412560 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 20026a988f644f559ef967bcf9b46ed7 because the tablet was not found. No further retry: Not found: Tablet not found: 20026a988f644f559ef967bcf9b46ed7
I20250625 13:51:51.412541 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea (DEFAULT_TABLE table=default.Table_d64bbacfddeb4945b2396e41e59cbb23 [id=f00b1eb97d6245a8b22ad21216ed5230]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:51.412763 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 969dc4a9b811496aa3d926a3ae3965ea because the tablet was not found. No further retry: Not found: Tablet not found: 969dc4a9b811496aa3d926a3ae3965ea
W20250625 13:51:51.412935 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 07458e047b88493e9e52423b4e85b091 because the tablet was not found. No further retry: Not found: Tablet not found: 07458e047b88493e9e52423b4e85b091
I20250625 13:51:51.413455 891 tablet_service.cc:1468] Processing CreateTablet for tablet ecd6be619b79429b956aae4a1bf1589d (DEFAULT_TABLE table=default.Table_f0748f143cf540dbbe1d769b411ac975 [id=d298fec0d0c14fc297e4a3b4f82b007a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.414662 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d92711284e64f71b31666800c250eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.417749 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.418249 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 20026a988f644f559ef967bcf9b46ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.418715 890 tablet_service.cc:1468] Processing CreateTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f (DEFAULT_TABLE table=default.Table_20b0b8fa5c384bbcb0fb51c0ecfd6a59 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.419078 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 07458e047b88493e9e52423b4e85b091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.419255 886 tablet_service.cc:1515] Processing DeleteTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.419494 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd6be619b79429b956aae4a1bf1589d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.420990 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f (DEFAULT_TABLE table=default.Table_20b0b8fa5c384bbcb0fb51c0ecfd6a59 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.421284 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f (DEFAULT_TABLE table=default.Table_755b00bf52dc402aaa1893a86293d177 [id=d47dd54b4d984cc1874992118033256d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.422585 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969dc4a9b811496aa3d926a3ae3965ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.424626 889 tablet_service.cc:1468] Processing CreateTablet for tablet 20026a988f644f559ef967bcf9b46ed7 (DEFAULT_TABLE table=default.Table_a13af807e80443588d6008d9ce3d727c [id=e88514d0a01b411ba21f2fd2b4d615e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.426111 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab41f8d02dff471ba272dbf7620b9c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.432076 888 tablet_service.cc:1468] Processing CreateTablet for tablet 07458e047b88493e9e52423b4e85b091 (DEFAULT_TABLE table=default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd [id=aa9bb127a9054fe18cee9dded1f88cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.432526 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d455a2dec4304cf48ce29ceeb0700b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.433146 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d455a2dec4304cf48ce29ceeb0700b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.440321 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20026a988f644f559ef967bcf9b46ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.441741 1166 tablet_bootstrap.cc:654] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.443154 1166 log.cc:826] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Log is configured to *not* fsync() on all Append() calls
I20250625 13:51:51.443559 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5afade7a5d1447a98934e4d614075d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.446985 1170 tablet_bootstrap.cc:492] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.447628 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969dc4a9b811496aa3d926a3ae3965ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.452126 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07458e047b88493e9e52423b4e85b091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.468205 1170 tablet_bootstrap.cc:654] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.469672 1170 log.cc:826] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Log is configured to *not* fsync() on all Append() calls
I20250625 13:51:51.469708 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c32eb049fff54229a1db62dc389476aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.470556 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c32eb049fff54229a1db62dc389476aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.472409 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9a166de4214d4318bd2b9aea4eed287c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.473224 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a166de4214d4318bd2b9aea4eed287c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.478415 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9875a118a9a74b7486cc83256370ddfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.479074 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9875a118a9a74b7486cc83256370ddfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.479655 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e42b0b5035ee4e30a0909702d04b052f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.480317 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e42b0b5035ee4e30a0909702d04b052f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.481877 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1de7d7247c89458280f07b46baed5d38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.482359 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1de7d7247c89458280f07b46baed5d38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.485042 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 07458e047b88493e9e52423b4e85b091 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.489356 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 07458e047b88493e9e52423b4e85b091 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.489784 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_85c33778738445ae9981ea2f1902e0ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.490334 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85c33778738445ae9981ea2f1902e0ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.493378 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1fe0d8f47f6240cca9142f4fda013f5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.494055 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fe0d8f47f6240cca9142f4fda013f5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.495677 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f4464e7b5bb640d18e585ce4383731c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.496289 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4464e7b5bb640d18e585ce4383731c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.497582 596 master_service.cc:948] ReplaceTablet: received request to replace tablet ecd6be619b79429b956aae4a1bf1589d from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.497993 591 master_service.cc:948] ReplaceTablet: received request to replace tablet a5afade7a5d1447a98934e4d614075d8 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.497920 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3f5b9820585e4ed89aa7d0abca37a52c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.498447 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f5b9820585e4ed89aa7d0abca37a52c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.503357 1170 tablet_bootstrap.cc:492] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.503701 1170 ts_tablet_manager.cc:1397] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.021s
I20250625 13:51:51.507831 594 master_service.cc:948] ReplaceTablet: received request to replace tablet d455a2dec4304cf48ce29ceeb0700b0f from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.510064 1170 raft_consensus.cc:357] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.513911 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_755b00bf52dc402aaa1893a86293d177" } new_table_name: "default.Table_4c9213ada50e4b4089e8d20f31fd18e9" modify_external_catalogs: true
I20250625 13:51:51.514338 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_34aa268b1b5c4293838192199719d473"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.514940 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34aa268b1b5c4293838192199719d473 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.510474 1170 raft_consensus.cc:383] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.516883 1170 raft_consensus.cc:738] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.518221 1170 consensus_queue.cc:260] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.519744 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20b0b8fa5c384bbcb0fb51c0ecfd6a59" } new_table_name: "default.Table_72ad58769a804f88b2ae43dd40dbf4d3" modify_external_catalogs: true
I20250625 13:51:51.528838 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_95be83c153474705b223639500ba9ff3" } new_table_name: "default.Table_4fee7c6bcd06448ab2794ada6f5d2804" modify_external_catalogs: true
I20250625 13:51:51.529857 1096 heartbeater.cc:499] Master 127.0.104.60:43307 was elected leader, sending a full tablet report...
I20250625 13:51:51.530910 1170 ts_tablet_manager.cc:1428] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.027s user 0.007s sys 0.006s
I20250625 13:51:51.531124 1171 tablet_bootstrap.cc:492] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.533447 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu with pid 424
W20250625 13:51:51.536252 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 20026a988f644f559ef967bcf9b46ed7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20026a988f644f559ef967bcf9b46ed7 already in progress: creating tablet
W20250625 13:51:51.536999 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 07458e047b88493e9e52423b4e85b091 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07458e047b88493e9e52423b4e85b091 already in progress: creating tablet
W20250625 13:51:51.537555 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 already in progress: creating tablet
W20250625 13:51:51.538385 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 969dc4a9b811496aa3d926a3ae3965ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 969dc4a9b811496aa3d926a3ae3965ea already in progress: creating tablet
I20250625 13:51:51.541028 1171 tablet_bootstrap.cc:654] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.541008 1173 raft_consensus.cc:491] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:51.541296 1173 raft_consensus.cc:513] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.542515 1171 log.cc:826] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Log is configured to *not* fsync() on all Append() calls
I20250625 13:51:51.543469 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7533c0ca831e480380ee740fa1ec802c" } new_table_name: "default.Table_fe20bac443bb450183cc5a10cdffff4c" modify_external_catalogs: true
I20250625 13:51:51.544183 1170 tablet_bootstrap.cc:492] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.545014 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9a166de4214d4318bd2b9aea4eed287c" } new_table_name: "default.Table_22c0e04fd41c41d981e66e94d5456772" modify_external_catalogs: true
I20250625 13:51:51.545739 885 tablet_service.cc:1468] Processing CreateTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 (DEFAULT_TABLE table=default.Table_c32eb049fff54229a1db62dc389476aa [id=bb31d8b38dd4479694c08744f757ef7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.546310 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7906d7c2fb541dcb1ec8e8dcc729c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.546443 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9875a118a9a74b7486cc83256370ddfd" } modify_external_catalogs: true
I20250625 13:51:51.546593 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9875a118a9a74b7486cc83256370ddfd" } modify_external_catalogs: true
I20250625 13:51:51.548039 1170 tablet_bootstrap.cc:654] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.549065 757 tablet_service.cc:1468] Processing CreateTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 (DEFAULT_TABLE table=default.Table_c32eb049fff54229a1db62dc389476aa [id=bb31d8b38dd4479694c08744f757ef7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.549454 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7906d7c2fb541dcb1ec8e8dcc729c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.550323 1023 tablet_service.cc:1468] Processing CreateTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 (DEFAULT_TABLE table=default.Table_c32eb049fff54229a1db62dc389476aa [id=bb31d8b38dd4479694c08744f757ef7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.550730 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7906d7c2fb541dcb1ec8e8dcc729c25. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:51.561780 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab41f8d02dff471ba272dbf7620b9c6f consensus state has no leader
I20250625 13:51:51.559383 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab41f8d02dff471ba272dbf7620b9c6f" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:51.563392 983 leader_election.cc:343] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.563843 1173 leader_election.cc:290] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:51.565694 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1de7d7247c89458280f07b46baed5d38" } modify_external_catalogs: true
I20250625 13:51:51.563480 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab41f8d02dff471ba272dbf7620b9c6f" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:51.566053 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1de7d7247c89458280f07b46baed5d38" } modify_external_catalogs: true
W20250625 13:51:51.567492 985 leader_election.cc:343] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.567695 985 leader_election.cc:304] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:51.568440 1173 raft_consensus.cc:2747] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:51.572862 1170 tablet_bootstrap.cc:492] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.572907 1171 tablet_bootstrap.cc:492] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:51.573037 1170 ts_tablet_manager.cc:1397] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250625 13:51:51.573207 1171 ts_tablet_manager.cc:1397] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.012s
I20250625 13:51:51.573985 1170 raft_consensus.cc:357] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:51.574190 1170 raft_consensus.cc:383] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.574265 1170 raft_consensus.cc:738] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.574537 1170 consensus_queue.cc:260] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:51.574965 1170 ts_tablet_manager.cc:1428] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250625 13:51:51.575973 1170 tablet_bootstrap.cc:492] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.576574 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ecd6be619b79429b956aae4a1bf1589d
I20250625 13:51:51.577100 596 catalog_manager.cc:6652] ReplaceTablet: tablet ecd6be619b79429b956aae4a1bf1589d deleted and replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2
I20250625 13:51:51.577415 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c32eb049fff54229a1db62dc389476aa" } modify_external_catalogs: true
I20250625 13:51:51.577575 1170 tablet_bootstrap.cc:654] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.577610 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c32eb049fff54229a1db62dc389476aa" } modify_external_catalogs: true
I20250625 13:51:51.578596 885 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd6be619b79429b956aae4a1bf1589d with delete_type TABLET_DATA_DELETED (replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.579093 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd6be619b79429b956aae4a1bf1589d with delete_type TABLET_DATA_DELETED (replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:51.579335 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ecd6be619b79429b956aae4a1bf1589d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecd6be619b79429b956aae4a1bf1589d already in progress: creating tablet
W20250625 13:51:51.579757 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ecd6be619b79429b956aae4a1bf1589d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecd6be619b79429b956aae4a1bf1589d already in progress: creating tablet
I20250625 13:51:51.579651 1171 raft_consensus.cc:357] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.579998 1171 raft_consensus.cc:383] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.580116 1171 raft_consensus.cc:738] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:51:51.580415 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4b212f7c1ab48d2807a9eebeec31c07 has no consensus state
I20250625 13:51:51.580785 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f0748f143cf540dbbe1d769b411ac975" } modify_external_catalogs: true
I20250625 13:51:51.580821 1171 consensus_queue.cc:260] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.580931 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f0748f143cf540dbbe1d769b411ac975" } modify_external_catalogs: true
I20250625 13:51:51.582988 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd6be619b79429b956aae4a1bf1589d with delete_type TABLET_DATA_DELETED (replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:51.583753 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ecd6be619b79429b956aae4a1bf1589d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecd6be619b79429b956aae4a1bf1589d already in progress: creating tablet
I20250625 13:51:51.588291 1170 tablet_bootstrap.cc:492] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.588462 1170 ts_tablet_manager.cc:1397] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20250625 13:51:51.588902 1171 ts_tablet_manager.cc:1428] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.016s user 0.002s sys 0.011s
I20250625 13:51:51.589264 1170 raft_consensus.cc:357] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.589455 1170 raft_consensus.cc:383] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.589546 1170 raft_consensus.cc:738] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.589766 1170 consensus_queue.cc:260] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.590346 1170 ts_tablet_manager.cc:1428] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:51.590687 1170 tablet_bootstrap.cc:492] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.592391 1170 tablet_bootstrap.cc:654] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.594002 1171 tablet_bootstrap.cc:492] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.595978 1171 tablet_bootstrap.cc:654] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.596976 588 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet faede735393c44b7a593dfa88985daef
W20250625 13:51:51.598955 1095 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:33015 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.62:33015 (error 108)
W20250625 13:51:51.602075 849 connection.cc:537] client connection to 127.0.104.62:33015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:51:51.602461 1170 tablet_bootstrap.cc:492] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.602623 1170 ts_tablet_manager.cc:1397] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
W20250625 13:51:51.602636 961 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:33015 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:51:51.604091 581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.62:33015 (error 108)
I20250625 13:51:51.604056 1170 raft_consensus.cc:357] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.604292 1170 raft_consensus.cc:383] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.604290 1166 tablet_bootstrap.cc:492] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:51.604373 1170 raft_consensus.cc:738] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.604626 1166 ts_tablet_manager.cc:1397] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.225s user 0.021s sys 0.015s
I20250625 13:51:51.604633 1170 consensus_queue.cc:260] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.606294 1170 ts_tablet_manager.cc:1428] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:51.606657 1170 tablet_bootstrap.cc:492] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.608851 1170 tablet_bootstrap.cc:654] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:51.611967 1099 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 13:51:51.599558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:33015 (local address 127.0.104.3:42491)
0625 13:51:51.599849 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0625 13:51:51.599874 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0625 13:51:51.600034 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 13:51:51.610419 (+ 10385us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145}
I20250625 13:51:51.613265 757 tablet_service.cc:1468] Processing CreateTablet for tablet 76f074156559406c875e65796e6b5ee6 (DEFAULT_TABLE table=default.Table_1fe0d8f47f6240cca9142f4fda013f5c [id=8c674ba68eb74eb8b6646ff2ef9c1561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.613797 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f074156559406c875e65796e6b5ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.615649 752 tablet_service.cc:1468] Processing CreateTablet for tablet 96d491878b8343be93892aa594ca9da7 (DEFAULT_TABLE table=default.Table_e42b0b5035ee4e30a0909702d04b052f [id=a364d50ed707453181496e70e26984e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.616042 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96d491878b8343be93892aa594ca9da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.619632 753 tablet_service.cc:1468] Processing CreateTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 (DEFAULT_TABLE table=default.Table_9875a118a9a74b7486cc83256370ddfd [id=72787a6701a64f08b90a8954cca4f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.620087 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbeb984fadf4a0ebc50087c1a921f07. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:51.623106 1098 tablet.cc:2378] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:51:51.623704 755 tablet_service.cc:1468] Processing CreateTablet for tablet 92a90b0dc85844a892ab247ae0d09a12 (DEFAULT_TABLE table=default.Table_85c33778738445ae9981ea2f1902e0ec [id=caf5ae6d8d9c482491f4b6d21c421f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.624118 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a90b0dc85844a892ab247ae0d09a12. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:51.626292 1181 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 13:51:51.603275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:33015 (local address 127.0.104.2:58723)
0625 13:51:51.624924 (+ 21649us) negotiation.cc:107] Waiting for socket to connect
0625 13:51:51.625177 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":21272,"thread_start_us":21364,"threads_started":1}
W20250625 13:51:51.627259 964 tablet.cc:2378] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:51:51.627751 758 tablet_service.cc:1468] Processing CreateTablet for tablet b4b212f7c1ab48d2807a9eebeec31c07 (DEFAULT_TABLE table=default.Table_22c0e04fd41c41d981e66e94d5456772 [id=9b25f559cfc44e418b075045f83e64f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.628054 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b212f7c1ab48d2807a9eebeec31c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.636107 1170 tablet_bootstrap.cc:492] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.636340 1170 ts_tablet_manager.cc:1397] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.003s
I20250625 13:51:51.637451 1170 raft_consensus.cc:357] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.637705 1170 raft_consensus.cc:383] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.637830 1170 raft_consensus.cc:738] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.638095 1170 consensus_queue.cc:260] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.638068 1166 raft_consensus.cc:357] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.638455 1166 raft_consensus.cc:383] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.638568 1166 raft_consensus.cc:738] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.639392 1166 consensus_queue.cc:260] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.641822 1170 ts_tablet_manager.cc:1428] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 13:51:51.646078 1170 tablet_bootstrap.cc:492] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.648173 1166 ts_tablet_manager.cc:1428] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.043s user 0.010s sys 0.004s
I20250625 13:51:51.648612 1170 tablet_bootstrap.cc:654] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.650374 1171 tablet_bootstrap.cc:492] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:51.650539 1171 ts_tablet_manager.cc:1397] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.000s
I20250625 13:51:51.651572 1171 raft_consensus.cc:357] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.651758 1171 raft_consensus.cc:383] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.651861 1171 raft_consensus.cc:738] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.652113 1171 consensus_queue.cc:260] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.652658 1171 ts_tablet_manager.cc:1428] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:51.652952 1171 tablet_bootstrap.cc:492] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.654963 1171 tablet_bootstrap.cc:654] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.661731 1170 tablet_bootstrap.cc:492] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.662067 1170 ts_tablet_manager.cc:1397] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20250625 13:51:51.663179 1170 raft_consensus.cc:357] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.663538 1170 raft_consensus.cc:383] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.663718 1170 raft_consensus.cc:738] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.664134 1170 consensus_queue.cc:260] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.667038 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3dbeb984fadf4a0ebc50087c1a921f07
I20250625 13:51:51.667987 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e42b0b5035ee4e30a0909702d04b052f" } modify_external_catalogs: true
I20250625 13:51:51.668210 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e42b0b5035ee4e30a0909702d04b052f" } modify_external_catalogs: true
I20250625 13:51:51.669437 1170 ts_tablet_manager.cc:1428] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 13:51:51.669852 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:51.671372 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3dbeb984fadf4a0ebc50087c1a921f07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dbeb984fadf4a0ebc50087c1a921f07 already in progress: creating tablet
I20250625 13:51:51.674165 1170 tablet_bootstrap.cc:492] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.677551 1170 tablet_bootstrap.cc:654] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:51.677626 830 tablet.cc:2378] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:51:51.681237 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 96d491878b8343be93892aa594ca9da7
I20250625 13:51:51.682020 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 96d491878b8343be93892aa594ca9da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:51.682832 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 96d491878b8343be93892aa594ca9da7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96d491878b8343be93892aa594ca9da7 already in progress: creating tablet
I20250625 13:51:51.685395 1171 tablet_bootstrap.cc:492] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:51.685631 1171 ts_tablet_manager.cc:1397] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.012s
I20250625 13:51:51.685940 753 tablet_service.cc:1468] Processing CreateTablet for tablet a418d2a6c8f24043a820967489c87e23 (DEFAULT_TABLE table=default.Table_34aa268b1b5c4293838192199719d473 [id=1ab84062698540c38a1047d6ce6476da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.686326 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a418d2a6c8f24043a820967489c87e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.686679 1171 raft_consensus.cc:357] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.686926 1171 raft_consensus.cc:383] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.687028 1171 raft_consensus.cc:738] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.687330 1171 consensus_queue.cc:260] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.687866 1171 ts_tablet_manager.cc:1428] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250625 13:51:51.688155 1171 tablet_bootstrap.cc:492] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.689837 1171 tablet_bootstrap.cc:654] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.692008 1170 tablet_bootstrap.cc:492] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.692168 1170 ts_tablet_manager.cc:1397] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250625 13:51:51.693073 1170 raft_consensus.cc:357] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.693271 1170 raft_consensus.cc:383] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.693387 1170 raft_consensus.cc:738] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.693648 1170 consensus_queue.cc:260] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.695888 1170 ts_tablet_manager.cc:1428] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:51:51.697045 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d455a2dec4304cf48ce29ceeb0700b0f
I20250625 13:51:51.697439 594 catalog_manager.cc:6652] ReplaceTablet: tablet d455a2dec4304cf48ce29ceeb0700b0f deleted and replaced by tablet a17f862a8a584047a8f0220f2eced869
I20250625 13:51:51.697872 1170 tablet_bootstrap.cc:492] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.698199 1166 tablet_bootstrap.cc:492] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:51.698220 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e7906d7c2fb541dcb1ec8e8dcc729c25
I20250625 13:51:51.698855 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 969dc4a9b811496aa3d926a3ae3965ea from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.700124 1170 tablet_bootstrap.cc:654] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.700284 590 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6d92711284e64f71b31666800c250eab (table default.Table_7533c0ca831e480380ee740fa1ec802c [id=96be558f5b694f3483d2501659dc5097]). Expected version 1 got 0
I20250625 13:51:51.700539 590 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet ab41f8d02dff471ba272dbf7620b9c6f (table default.Table_4c9213ada50e4b4089e8d20f31fd18e9 [id=d47dd54b4d984cc1874992118033256d]). Expected version 2 got 0
W20250625 13:51:51.701201 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a17f862a8a584047a8f0220f2eced869 has no consensus state
I20250625 13:51:51.701608 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c94d10947ecf49fba6f7608086ef35bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.702207 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c94d10947ecf49fba6f7608086ef35bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:51.703373 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d92711284e64f71b31666800c250eab consensus state has no leader
I20250625 13:51:51.703431 1166 tablet_bootstrap.cc:654] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:51.703733 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0723e91650174f5badd74f7756c5a54e has no consensus state
I20250625 13:51:51.703986 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet b15fb962c53f46f890d8f11cae6eb4a2
I20250625 13:51:51.705523 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a5afade7a5d1447a98934e4d614075d8
I20250625 13:51:51.706040 591 catalog_manager.cc:6652] ReplaceTablet: tablet a5afade7a5d1447a98934e4d614075d8 deleted and replaced by tablet 0723e91650174f5badd74f7756c5a54e
I20250625 13:51:51.707893 753 tablet_service.cc:1468] Processing CreateTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c (DEFAULT_TABLE table=default.Table_3f5b9820585e4ed89aa7d0abca37a52c [id=d7e8d11b88e240b89d40754b67570894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.708316 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b44c85b12fe64fc99fcf0a79a5909c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.710359 752 tablet_service.cc:1515] Processing DeleteTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f with delete_type TABLET_DATA_DELETED (replaced by tablet a17f862a8a584047a8f0220f2eced869) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.710628 755 tablet_service.cc:1515] Processing DeleteTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.710999 757 tablet_service.cc:1468] Processing CreateTablet for tablet 3c282591e90c40948563fda06453e4d4 (DEFAULT_TABLE table=default.Table_f4464e7b5bb640d18e585ce4383731c9 [id=18ba0d9199d14289bed69ea544d3dced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.711198 752 tablet_service.cc:1515] Processing DeleteTablet for tablet a5afade7a5d1447a98934e4d614075d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0723e91650174f5badd74f7756c5a54e) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.712025 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c282591e90c40948563fda06453e4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.714978 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8778e702d2c74eacbef8d32210bc05ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.715696 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8778e702d2c74eacbef8d32210bc05ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:51.717657 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d455a2dec4304cf48ce29ceeb0700b0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d455a2dec4304cf48ce29ceeb0700b0f already in progress: creating tablet
W20250625 13:51:51.717929 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7906d7c2fb541dcb1ec8e8dcc729c25 already in progress: creating tablet
W20250625 13:51:51.718119 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a5afade7a5d1447a98934e4d614075d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5afade7a5d1447a98934e4d614075d8 already in progress: creating tablet
I20250625 13:51:51.718689 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 20026a988f644f559ef967bcf9b46ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.718992 1165 tablet_replica.cc:331] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:51.719539 1165 raft_consensus.cc:2241] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:51.719987 1165 raft_consensus.cc:2270] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.722584 1165 ts_tablet_manager.cc:1905] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.724236 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_194007e5294849f9a6734e39b20d4fff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.724997 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3d4e8fd3ed9144f2ba1aeb2cdea115e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.725817 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2523340d8092470691c4d98f5df6961e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.729635 1165 ts_tablet_manager.cc:1918] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.733944 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1a827c72123c47dda8cb83ee349de91c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.735769 1165 log.cc:1199] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/20026a988f644f559ef967bcf9b46ed7
I20250625 13:51:51.736274 1165 ts_tablet_manager.cc:1939] T 20026a988f644f559ef967bcf9b46ed7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:51:51.742033 823 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:33015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111)
I20250625 13:51:51.745961 1170 tablet_bootstrap.cc:492] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.746215 1170 ts_tablet_manager.cc:1397] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.017s
I20250625 13:51:51.746843 1171 tablet_bootstrap.cc:492] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:51.738062 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 20026a988f644f559ef967bcf9b46ed7 (table default.Table_a13af807e80443588d6008d9ce3d727c [id=e88514d0a01b411ba21f2fd2b4d615e5]) successfully deleted
I20250625 13:51:51.747754 1170 raft_consensus.cc:357] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:51.743417 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_194007e5294849f9a6734e39b20d4fff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.748129 1170 raft_consensus.cc:383] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.748152 1173 raft_consensus.cc:491] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:51.748242 1170 raft_consensus.cc:738] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.748313 1173 raft_consensus.cc:513] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.748508 1170 consensus_queue.cc:260] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.748946 1173 leader_election.cc:290] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:51.749173 1170 ts_tablet_manager.cc:1428] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:51:51.749491 1170 tablet_bootstrap.cc:492] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:51:51.743487 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d4e8fd3ed9144f2ba1aeb2cdea115e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.750468 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5afade7a5d1447a98934e4d614075d8" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:51.751034 985 leader_election.cc:343] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:51.743538 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2523340d8092470691c4d98f5df6961e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.751227 1170 tablet_bootstrap.cc:654] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.751780 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5afade7a5d1447a98934e4d614075d8" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:51.743579 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a827c72123c47dda8cb83ee349de91c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:51.753810 983 leader_election.cc:343] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.754011 983 leader_election.cc:304] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:51.754797 1173 raft_consensus.cc:2747] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:51.755188 1171 ts_tablet_manager.cc:1397] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.017s
W20250625 13:51:51.743618 581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:33015, user_credentials={real_user=slave}} blocked reactor thread for 139535us
I20250625 13:51:51.756491 1171 raft_consensus.cc:357] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.756888 1171 raft_consensus.cc:383] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.757066 1171 raft_consensus.cc:738] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.757486 1171 consensus_queue.cc:260] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.758139 885 tablet_service.cc:1468] Processing CreateTablet for tablet 76f074156559406c875e65796e6b5ee6 (DEFAULT_TABLE table=default.Table_1fe0d8f47f6240cca9142f4fda013f5c [id=8c674ba68eb74eb8b6646ff2ef9c1561]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:51.758003 581 consensus_peers.cc:487] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 -> Peer 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015): Couldn't send request to peer 3fec9d8a79054a5991ae90d8a3101e32. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:51.758569 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f074156559406c875e65796e6b5ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.737624 593 master_service.cc:948] ReplaceTablet: received request to replace tablet ecd6be619b79429b956aae4a1bf1589d from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.759716 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 96d491878b8343be93892aa594ca9da7 (DEFAULT_TABLE table=default.Table_e42b0b5035ee4e30a0909702d04b052f [id=a364d50ed707453181496e70e26984e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.762090 896 tablet_service.cc:1468] Processing CreateTablet for tablet 96d491878b8343be93892aa594ca9da7 (DEFAULT_TABLE table=default.Table_e42b0b5035ee4e30a0909702d04b052f [id=a364d50ed707453181496e70e26984e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.762463 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96d491878b8343be93892aa594ca9da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.762506 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e02e9aad565644299f4613c5b807162c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:51.763082 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e02e9aad565644299f4613c5b807162c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.763654 889 tablet_service.cc:1468] Processing CreateTablet for tablet b4b212f7c1ab48d2807a9eebeec31c07 (DEFAULT_TABLE table=default.Table_22c0e04fd41c41d981e66e94d5456772 [id=9b25f559cfc44e418b075045f83e64f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.763988 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b212f7c1ab48d2807a9eebeec31c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.764204 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 92a90b0dc85844a892ab247ae0d09a12 (DEFAULT_TABLE table=default.Table_85c33778738445ae9981ea2f1902e0ec [id=caf5ae6d8d9c482491f4b6d21c421f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.765255 1171 ts_tablet_manager.cc:1428] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250625 13:51:51.765664 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 (DEFAULT_TABLE table=default.Table_9875a118a9a74b7486cc83256370ddfd [id=72787a6701a64f08b90a8954cca4f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.764804 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 76f074156559406c875e65796e6b5ee6 (DEFAULT_TABLE table=default.Table_1fe0d8f47f6240cca9142f4fda013f5c [id=8c674ba68eb74eb8b6646ff2ef9c1561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.766054 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b4b212f7c1ab48d2807a9eebeec31c07 (DEFAULT_TABLE table=default.Table_22c0e04fd41c41d981e66e94d5456772 [id=9b25f559cfc44e418b075045f83e64f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.766278 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96d491878b8343be93892aa594ca9da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.767576 891 tablet_service.cc:1468] Processing CreateTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 (DEFAULT_TABLE table=default.Table_9875a118a9a74b7486cc83256370ddfd [id=72787a6701a64f08b90a8954cca4f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.767781 1171 tablet_bootstrap.cc:492] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.767927 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbeb984fadf4a0ebc50087c1a921f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.770628 1171 tablet_bootstrap.cc:654] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.771380 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.771590 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a90b0dc85844a892ab247ae0d09a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.771620 895 tablet_service.cc:1468] Processing CreateTablet for tablet 92a90b0dc85844a892ab247ae0d09a12 (DEFAULT_TABLE table=default.Table_85c33778738445ae9981ea2f1902e0ec [id=caf5ae6d8d9c482491f4b6d21c421f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.771974 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92a90b0dc85844a892ab247ae0d09a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.775436 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 96d491878b8343be93892aa594ca9da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.775830 893 tablet_service.cc:1468] Processing CreateTablet for tablet a418d2a6c8f24043a820967489c87e23 (DEFAULT_TABLE table=default.Table_34aa268b1b5c4293838192199719d473 [id=1ab84062698540c38a1047d6ce6476da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.776175 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a418d2a6c8f24043a820967489c87e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.779179 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f074156559406c875e65796e6b5ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.779491 892 tablet_service.cc:1468] Processing CreateTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c (DEFAULT_TABLE table=default.Table_3f5b9820585e4ed89aa7d0abca37a52c [id=d7e8d11b88e240b89d40754b67570894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.779826 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b44c85b12fe64fc99fcf0a79a5909c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.779940 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b212f7c1ab48d2807a9eebeec31c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.780612 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dbeb984fadf4a0ebc50087c1a921f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.782459 1030 tablet_service.cc:1468] Processing CreateTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c (DEFAULT_TABLE table=default.Table_3f5b9820585e4ed89aa7d0abca37a52c [id=d7e8d11b88e240b89d40754b67570894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.782925 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b44c85b12fe64fc99fcf0a79a5909c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.783109 890 tablet_service.cc:1515] Processing DeleteTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f with delete_type TABLET_DATA_DELETED (replaced by tablet a17f862a8a584047a8f0220f2eced869) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.783568 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3c282591e90c40948563fda06453e4d4 (DEFAULT_TABLE table=default.Table_f4464e7b5bb640d18e585ce4383731c9 [id=18ba0d9199d14289bed69ea544d3dced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.783733 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.784071 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c282591e90c40948563fda06453e4d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.785532 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f with delete_type TABLET_DATA_DELETED (replaced by tablet a17f862a8a584047a8f0220f2eced869) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.786024 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a418d2a6c8f24043a820967489c87e23 (DEFAULT_TABLE table=default.Table_34aa268b1b5c4293838192199719d473 [id=1ab84062698540c38a1047d6ce6476da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.786465 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a418d2a6c8f24043a820967489c87e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.787338 1167 tablet_replica.cc:331] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:51.787366 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 20026a988f644f559ef967bcf9b46ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.787635 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a5afade7a5d1447a98934e4d614075d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0723e91650174f5badd74f7756c5a54e) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.787591 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_70ae19ea7742453484d9525b11f16752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.787998 1167 raft_consensus.cc:2241] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:51.788228 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70ae19ea7742453484d9525b11f16752 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.788503 1168 tablet_replica.cc:331] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:51.788658 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet a5afade7a5d1447a98934e4d614075d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0723e91650174f5badd74f7756c5a54e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.788827 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 96d491878b8343be93892aa594ca9da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.788944 1168 raft_consensus.cc:2241] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:51.789179 1167 raft_consensus.cc:2270] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.789304 1168 raft_consensus.cc:2270] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.790616 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.790788 887 tablet_service.cc:1515] Processing DeleteTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.792550 1168 ts_tablet_manager.cc:1905] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.794822 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd6be619b79429b956aae4a1bf1589d with delete_type TABLET_DATA_DELETED (replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:51.794888 581 catalog_manager.cc:4895] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 96d491878b8343be93892aa594ca9da7 because the tablet was not found. No further retry: Not found: Tablet not found: 96d491878b8343be93892aa594ca9da7
I20250625 13:51:51.795089 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 20026a988f644f559ef967bcf9b46ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.794075 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 3c282591e90c40948563fda06453e4d4 (DEFAULT_TABLE table=default.Table_f4464e7b5bb640d18e585ce4383731c9 [id=18ba0d9199d14289bed69ea544d3dced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.797550 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c282591e90c40948563fda06453e4d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:51.800750 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3dbeb984fadf4a0ebc50087c1a921f07 because the tablet was not found. No further retry: Not found: Tablet not found: 3dbeb984fadf4a0ebc50087c1a921f07
I20250625 13:51:51.801187 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.801973 1174 raft_consensus.cc:491] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:51.802129 1174 raft_consensus.cc:513] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:51.802302 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7906d7c2fb541dcb1ec8e8dcc729c25 already in progress: creating tablet
I20250625 13:51:51.802006 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:51.803254 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4e029acc913b4ce5b29a1fa12208e2cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.803442 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 07458e047b88493e9e52423b4e85b091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:51.803808 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e029acc913b4ce5b29a1fa12208e2cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.803882 1167 ts_tablet_manager.cc:1905] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.804111 1174 leader_election.cc:290] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:51.804761 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07458e047b88493e9e52423b4e85b091" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:51.805310 983 leader_election.cc:343] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.806073 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07458e047b88493e9e52423b4e85b091" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:51.806633 985 leader_election.cc:343] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.806771 985 leader_election.cc:304] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:51.807200 1174 raft_consensus.cc:2747] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:51.795220 887 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd6be619b79429b956aae4a1bf1589d with delete_type TABLET_DATA_DELETED (replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:51.812487 757 tablet_service.cc:1468] Processing CreateTablet for tablet a17f862a8a584047a8f0220f2eced869 (DEFAULT_TABLE table=default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.812914 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17f862a8a584047a8f0220f2eced869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.815621 753 tablet_service.cc:1468] Processing CreateTablet for tablet 0723e91650174f5badd74f7756c5a54e (DEFAULT_TABLE table=default.Table_4fee7c6bcd06448ab2794ada6f5d2804 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.816018 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0723e91650174f5badd74f7756c5a54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.820953 889 tablet_service.cc:1468] Processing CreateTablet for tablet a17f862a8a584047a8f0220f2eced869 (DEFAULT_TABLE table=default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.821337 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17f862a8a584047a8f0220f2eced869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.821379 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4fee7c6bcd06448ab2794ada6f5d2804" } new_table_name: "default.Table_4afe38f204684075b394ad74549a04c2" modify_external_catalogs: true
I20250625 13:51:51.823568 1020 tablet_service.cc:1468] Processing CreateTablet for tablet a17f862a8a584047a8f0220f2eced869 (DEFAULT_TABLE table=default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.824016 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17f862a8a584047a8f0220f2eced869. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.828395 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9cc4881bfac74cfab175b6711d7b1f83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:51.829725 896 tablet_service.cc:1468] Processing CreateTablet for tablet 0723e91650174f5badd74f7756c5a54e (DEFAULT_TABLE table=default.Table_4fee7c6bcd06448ab2794ada6f5d2804 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.830247 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0723e91650174f5badd74f7756c5a54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.801767 1173 raft_consensus.cc:491] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:51.831691 1173 raft_consensus.cc:513] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.832599 1173 leader_election.cc:290] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:51.834168 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cc4881bfac74cfab175b6711d7b1f83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:51.836758 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd6be619b79429b956aae4a1bf1589d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:51.837412 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd6be619b79429b956aae4a1bf1589d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:51.837857 916 raft_consensus.cc:2466] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:51.838744 983 leader_election.cc:304] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:51:51.839540 985 leader_election.cc:343] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.840106 1173 raft_consensus.cc:2802] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:51.840269 1173 raft_consensus.cc:491] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:51.840384 1173 raft_consensus.cc:3058] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:51.856685 1173 raft_consensus.cc:513] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.858294 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd6be619b79429b956aae4a1bf1589d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
W20250625 13:51:51.859095 985 leader_election.cc:343] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.860208 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd6be619b79429b956aae4a1bf1589d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:51.860455 916 raft_consensus.cc:3058] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:51.863565 1173 leader_election.cc:290] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:51.865832 916 raft_consensus.cc:2466] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:51.866477 983 leader_election.cc:304] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:51.872388 1166 tablet_bootstrap.cc:492] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:51.872638 1166 ts_tablet_manager.cc:1397] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.175s user 0.033s sys 0.023s
I20250625 13:51:51.873981 1166 raft_consensus.cc:357] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.874257 1166 raft_consensus.cc:383] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.874382 1166 raft_consensus.cc:738] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.874642 1166 consensus_queue.cc:260] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.875237 1166 ts_tablet_manager.cc:1428] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:51.875560 1166 tablet_bootstrap.cc:492] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:51.877462 1166 tablet_bootstrap.cc:654] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.880120 1167 ts_tablet_manager.cc:1918] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.880331 1167 log.cc:1199] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d455a2dec4304cf48ce29ceeb0700b0f
I20250625 13:51:51.880877 1167 ts_tablet_manager.cc:1939] T d455a2dec4304cf48ce29ceeb0700b0f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:51.881997 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f4464e7b5bb640d18e585ce4383731c9" } new_table_name: "default.Table_893b774d7d574f8e89a027212ab17b74" modify_external_catalogs: true
I20250625 13:51:51.882447 1168 ts_tablet_manager.cc:1918] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.882607 1168 log.cc:1199] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d455a2dec4304cf48ce29ceeb0700b0f
I20250625 13:51:51.882874 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet d455a2dec4304cf48ce29ceeb0700b0f (table default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]) successfully deleted
I20250625 13:51:51.883734 1167 tablet_replica.cc:331] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:51.883922 1167 raft_consensus.cc:2241] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:51.884121 1167 raft_consensus.cc:2270] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.885012 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 20026a988f644f559ef967bcf9b46ed7 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:51.885353 1167 ts_tablet_manager.cc:1905] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.885572 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3f5b9820585e4ed89aa7d0abca37a52c" } modify_external_catalogs: true
I20250625 13:51:51.885775 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3f5b9820585e4ed89aa7d0abca37a52c" } modify_external_catalogs: true
I20250625 13:51:51.886696 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_72ad58769a804f88b2ae43dd40dbf4d3" } modify_external_catalogs: true
I20250625 13:51:51.886883 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_72ad58769a804f88b2ae43dd40dbf4d3" } modify_external_catalogs: true
I20250625 13:51:51.887843 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:51.888598 1165 tablet_replica.cc:331] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:51.888798 1165 raft_consensus.cc:2241] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:51.888995 1165 raft_consensus.cc:2270] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.889556 1174 raft_consensus.cc:2802] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:51.889791 1165 ts_tablet_manager.cc:1905] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.890209 1166 tablet_bootstrap.cc:492] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:51.890420 1166 ts_tablet_manager.cc:1397] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20250625 13:51:51.890492 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:51.890682 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: LEADER
I20250625 13:51:51.891750 1166 raft_consensus.cc:357] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.892023 1166 raft_consensus.cc:383] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.892165 1166 raft_consensus.cc:738] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.892469 1166 consensus_queue.cc:260] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.893080 1166 ts_tablet_manager.cc:1428] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:51.893457 1166 tablet_bootstrap.cc:492] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:51.893921 1173 raft_consensus.cc:491] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:51.894070 1173 raft_consensus.cc:513] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:51.895116 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c87280e6a3e84e6ba5b0a90e1e7ecec3" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:51.895335 1166 tablet_bootstrap.cc:654] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:51.895743 983 leader_election.cc:343] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.896965 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 0723e91650174f5badd74f7756c5a54e (DEFAULT_TABLE table=default.Table_4fee7c6bcd06448ab2794ada6f5d2804 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.897408 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0723e91650174f5badd74f7756c5a54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.898638 1167 ts_tablet_manager.cc:1918] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.898782 1167 log.cc:1199] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a5afade7a5d1447a98934e4d614075d8
I20250625 13:51:51.899154 1167 ts_tablet_manager.cc:1939] T a5afade7a5d1447a98934e4d614075d8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:51.899881 1167 tablet_replica.cc:331] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:51.899809 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4c9213ada50e4b4089e8d20f31fd18e9" } modify_external_catalogs: true
I20250625 13:51:51.900007 1173 leader_election.cc:290] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:51.899986 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4c9213ada50e4b4089e8d20f31fd18e9" } modify_external_catalogs: true
I20250625 13:51:51.900833 1165 ts_tablet_manager.cc:1918] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.901183 1165 log.cc:1199] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/969dc4a9b811496aa3d926a3ae3965ea
I20250625 13:51:51.901244 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c87280e6a3e84e6ba5b0a90e1e7ecec3" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:51.901796 985 leader_election.cc:343] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:51.901948 985 leader_election.cc:304] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:51.902384 1165 ts_tablet_manager.cc:1939] T 969dc4a9b811496aa3d926a3ae3965ea P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:51.902686 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a5afade7a5d1447a98934e4d614075d8 (table default.Table_4fee7c6bcd06448ab2794ada6f5d2804 [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
I20250625 13:51:51.902856 1170 tablet_bootstrap.cc:492] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:51:51.902979 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 96d491878b8343be93892aa594ca9da7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96d491878b8343be93892aa594ca9da7 already in progress: creating tablet
I20250625 13:51:51.903017 1170 ts_tablet_manager.cc:1397] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.154s user 0.041s sys 0.065s
I20250625 13:51:51.903357 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 969dc4a9b811496aa3d926a3ae3965ea (table default.Table_d64bbacfddeb4945b2396e41e59cbb23 [id=f00b1eb97d6245a8b22ad21216ed5230]) successfully deleted
I20250625 13:51:51.903576 1173 raft_consensus.cc:2747] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:51.904052 1170 raft_consensus.cc:357] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.904289 1170 raft_consensus.cc:383] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.904436 1170 raft_consensus.cc:738] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.904718 1170 consensus_queue.cc:260] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.905357 1170 ts_tablet_manager.cc:1428] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:51:51.905683 1170 tablet_bootstrap.cc:492] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.908229 1171 tablet_bootstrap.cc:492] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:51.908411 1171 ts_tablet_manager.cc:1397] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.141s user 0.001s sys 0.006s
I20250625 13:51:51.909513 1171 raft_consensus.cc:357] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.910248 1171 raft_consensus.cc:383] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.910529 1171 raft_consensus.cc:738] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.910957 1171 consensus_queue.cc:260] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.911572 1174 raft_consensus.cc:695] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:51.912185 1174 consensus_queue.cc:237] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.914358 1171 ts_tablet_manager.cc:1428] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250625 13:51:51.915683 1168 ts_tablet_manager.cc:1939] T d455a2dec4304cf48ce29ceeb0700b0f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:51.916175 1171 tablet_bootstrap.cc:492] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.917827 1168 tablet_replica.cc:331] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:51.918108 1168 raft_consensus.cc:2241] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:51.918108 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet d455a2dec4304cf48ce29ceeb0700b0f (table default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]) successfully deleted
I20250625 13:51:51.918311 1171 tablet_bootstrap.cc:654] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.918335 1168 raft_consensus.cc:2270] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.920361 1168 ts_tablet_manager.cc:1905] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.920665 1166 tablet_bootstrap.cc:492] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:51.920888 1166 ts_tablet_manager.cc:1397] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.008s
I20250625 13:51:51.921639 1166 raft_consensus.cc:357] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.921883 1166 raft_consensus.cc:383] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.921996 1166 raft_consensus.cc:738] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.922257 1166 consensus_queue.cc:260] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.924624 1166 ts_tablet_manager.cc:1428] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:51:51.925770 1166 tablet_bootstrap.cc:492] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:51.927134 1171 tablet_bootstrap.cc:492] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:51.927263 1170 tablet_bootstrap.cc:654] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.927349 1171 ts_tablet_manager.cc:1397] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20250625 13:51:51.928190 1171 raft_consensus.cc:357] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.928436 1171 raft_consensus.cc:383] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.928575 1171 raft_consensus.cc:738] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.928612 1166 tablet_bootstrap.cc:654] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.928933 1171 consensus_queue.cc:260] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.932340 1171 ts_tablet_manager.cc:1428] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250625 13:51:51.935278 1171 tablet_bootstrap.cc:492] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.936983 1171 tablet_bootstrap.cc:654] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.939798 1168 ts_tablet_manager.cc:1918] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.940490 1168 log.cc:1199] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/20026a988f644f559ef967bcf9b46ed7
I20250625 13:51:51.941028 1168 ts_tablet_manager.cc:1939] T 20026a988f644f559ef967bcf9b46ed7 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:51.941211 1170 tablet_bootstrap.cc:492] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:51.942451 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 20026a988f644f559ef967bcf9b46ed7 (table default.Table_a13af807e80443588d6008d9ce3d727c [id=e88514d0a01b411ba21f2fd2b4d615e5]) successfully deleted
I20250625 13:51:51.943817 1168 tablet_replica.cc:331] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:51.944154 1168 raft_consensus.cc:2241] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:51.944844 1168 raft_consensus.cc:2270] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.945557 1170 ts_tablet_manager.cc:1397] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.001s
I20250625 13:51:51.945827 1168 ts_tablet_manager.cc:1905] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.946991 1170 raft_consensus.cc:357] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.948544 1170 raft_consensus.cc:383] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.948905 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2523340d8092470691c4d98f5df6961e" } modify_external_catalogs: true
I20250625 13:51:51.949093 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2523340d8092470691c4d98f5df6961e" } modify_external_catalogs: true
I20250625 13:51:51.949434 1170 raft_consensus.cc:738] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.950089 1166 tablet_bootstrap.cc:492] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:51.950311 1166 ts_tablet_manager.cc:1397] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20250625 13:51:51.950373 1170 consensus_queue.cc:260] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.951253 1166 raft_consensus.cc:357] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.951653 1166 raft_consensus.cc:383] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.951720 1170 ts_tablet_manager.cc:1428] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:51:51.951788 1166 raft_consensus.cc:738] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.952216 1166 consensus_queue.cc:260] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.952459 1170 tablet_bootstrap.cc:492] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:51.953709 1171 tablet_bootstrap.cc:492] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:51.954236 1170 tablet_bootstrap.cc:654] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.954648 1166 ts_tablet_manager.cc:1428] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:51.953951 1171 ts_tablet_manager.cc:1397] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20250625 13:51:51.955479 1171 raft_consensus.cc:357] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.955753 1171 raft_consensus.cc:383] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.955864 1166 tablet_bootstrap.cc:492] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:51.955893 1171 raft_consensus.cc:738] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.956208 1171 consensus_queue.cc:260] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.958412 1166 tablet_bootstrap.cc:654] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.959234 1171 ts_tablet_manager.cc:1428] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:51.960938 1167 raft_consensus.cc:2241] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:51.963851 1167 pending_rounds.cc:70] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:51:51.964012 1167 pending_rounds.cc:77] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520560797777920 op_type: NO_OP noop_request { }
I20250625 13:51:51.964203 1167 raft_consensus.cc:2887] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:51.964305 1167 raft_consensus.cc:2270] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.967564 1171 tablet_bootstrap.cc:492] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:51.970062 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 (DEFAULT_TABLE table=default.Table_c94d10947ecf49fba6f7608086ef35bc [id=263e68a7d9344016afdb8f4dc5b27b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.970450 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872724cd95bf473cb40e5f8ed8b5bd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.970508 1171 tablet_bootstrap.cc:654] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:51.972623 1168 ts_tablet_manager.cc:1918] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.972734 1168 log.cc:1199] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a5afade7a5d1447a98934e4d614075d8
I20250625 13:51:51.972641 1166 tablet_bootstrap.cc:492] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:51.972921 1166 ts_tablet_manager.cc:1397] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20250625 13:51:51.973091 1168 ts_tablet_manager.cc:1939] T a5afade7a5d1447a98934e4d614075d8 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:51.973884 1166 raft_consensus.cc:357] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.974144 1166 raft_consensus.cc:383] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.974247 1166 raft_consensus.cc:738] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.974596 1166 consensus_queue.cc:260] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.976146 1168 tablet_replica.cc:331] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:51.976969 753 tablet_service.cc:1468] Processing CreateTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 (DEFAULT_TABLE table=default.Table_c94d10947ecf49fba6f7608086ef35bc [id=263e68a7d9344016afdb8f4dc5b27b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.977416 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872724cd95bf473cb40e5f8ed8b5bd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.978009 1168 raft_consensus.cc:2241] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:51.978165 1168 raft_consensus.cc:2270] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:51.978400 1166 ts_tablet_manager.cc:1428] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:51:51.979784 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet a5afade7a5d1447a98934e4d614075d8 (table default.Table_4fee7c6bcd06448ab2794ada6f5d2804 [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
I20250625 13:51:51.979342 1168 ts_tablet_manager.cc:1905] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:51.976146 896 tablet_service.cc:1468] Processing CreateTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 (DEFAULT_TABLE table=default.Table_c94d10947ecf49fba6f7608086ef35bc [id=263e68a7d9344016afdb8f4dc5b27b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:51.980422 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872724cd95bf473cb40e5f8ed8b5bd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:51.980952 1166 tablet_bootstrap.cc:492] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:51.983755 1166 tablet_bootstrap.cc:654] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:51.983866 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3dbeb984fadf4a0ebc50087c1a921f07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dbeb984fadf4a0ebc50087c1a921f07 already in progress: creating tablet
W20250625 13:51:51.984117 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7906d7c2fb541dcb1ec8e8dcc729c25 already in progress: creating tablet
I20250625 13:51:51.992393 1166 tablet_bootstrap.cc:492] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:51.992734 1166 ts_tablet_manager.cc:1397] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20250625 13:51:51.994060 1166 raft_consensus.cc:357] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.994519 1166 raft_consensus.cc:383] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:51.994668 1166 raft_consensus.cc:738] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:51.995173 1168 ts_tablet_manager.cc:1918] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:51.995108 1166 consensus_queue.cc:260] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:51.997740 1168 log.cc:1199] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ecd6be619b79429b956aae4a1bf1589d
I20250625 13:51:51.998176 1168 ts_tablet_manager.cc:1939] T ecd6be619b79429b956aae4a1bf1589d P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:51.998479 1166 ts_tablet_manager.cc:1428] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250625 13:51:52.000339 1166 tablet_bootstrap.cc:492] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.000384 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet ecd6be619b79429b956aae4a1bf1589d (table default.Table_f0748f143cf540dbbe1d769b411ac975 [id=d298fec0d0c14fc297e4a3b4f82b007a]) successfully deleted
I20250625 13:51:52.004492 1166 tablet_bootstrap.cc:654] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.005193 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8778e702d2c74eacbef8d32210bc05ea" } new_table_name: "default.Table_ffbe3938117d49e0b9627a5c8f1d45f4" modify_external_catalogs: true
I20250625 13:51:52.013083 1166 tablet_bootstrap.cc:492] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.013382 1166 ts_tablet_manager.cc:1397] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20250625 13:51:52.014526 1166 raft_consensus.cc:357] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.014853 1167 ts_tablet_manager.cc:1905] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.014860 1166 raft_consensus.cc:383] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.015081 1166 raft_consensus.cc:738] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.015473 1166 consensus_queue.cc:260] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.018507 1166 ts_tablet_manager.cc:1428] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:51:52.020012 1166 tablet_bootstrap.cc:492] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.023988 1166 tablet_bootstrap.cc:654] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.032526 1166 tablet_bootstrap.cc:492] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
W20250625 13:51:52.032734 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c282591e90c40948563fda06453e4d4 consensus state has no leader
I20250625 13:51:52.032912 1166 ts_tablet_manager.cc:1397] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20250625 13:51:52.034198 1166 raft_consensus.cc:357] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.034615 1166 raft_consensus.cc:383] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.034756 1166 raft_consensus.cc:738] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.035179 1166 consensus_queue.cc:260] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.036689 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fe20bac443bb450183cc5a10cdffff4c" } new_table_name: "default.Table_2273827d860d46af8da96df2ffd22a04" modify_external_catalogs: true
I20250625 13:51:52.038435 1166 ts_tablet_manager.cc:1428] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:51:52.039996 1184 raft_consensus.cc:491] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.040285 1184 raft_consensus.cc:513] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.041766 1167 ts_tablet_manager.cc:1918] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:52.041981 1167 log.cc:1199] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ecd6be619b79429b956aae4a1bf1589d
I20250625 13:51:52.042563 1167 ts_tablet_manager.cc:1939] T ecd6be619b79429b956aae4a1bf1589d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.042577 1184 leader_election.cc:290] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.043993 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ecd6be619b79429b956aae4a1bf1589d (table default.Table_f0748f143cf540dbbe1d769b411ac975 [id=d298fec0d0c14fc297e4a3b4f82b007a]) successfully deleted
I20250625 13:51:52.048936 593 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet fd138c77e9714366b3cc8a763480efd7
I20250625 13:51:52.059396 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1a827c72123c47dda8cb83ee349de91c" } new_table_name: "default.Table_05f514ab26624c719af36996404ef419" modify_external_catalogs: true
I20250625 13:51:52.059820 1184 raft_consensus.cc:491] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.060468 1184 raft_consensus.cc:513] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.061460 1184 leader_election.cc:290] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:51:52.067152 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15750915ff7746ffa66509184be55dc4 has no consensus state
I20250625 13:51:52.071422 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c94d10947ecf49fba6f7608086ef35bc" } modify_external_catalogs: true
I20250625 13:51:52.071717 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c94d10947ecf49fba6f7608086ef35bc" } modify_external_catalogs: true
I20250625 13:51:52.073814 1166 tablet_bootstrap.cc:492] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.075800 1167 tablet_replica.cc:331] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.079486 1166 tablet_bootstrap.cc:654] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.083141 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c87280e6a3e84e6ba5b0a90e1e7ecec3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:52.083802 711 leader_election.cc:343] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: d455a2dec4304cf48ce29ceeb0700b0f
W20250625 13:51:52.084170 711 leader_election.cc:343] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.077481 1170 tablet_bootstrap.cc:492] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:52.084517 1170 ts_tablet_manager.cc:1397] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.132s user 0.040s sys 0.069s
I20250625 13:51:52.084935 1167 raft_consensus.cc:2241] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.086568 1167 raft_consensus.cc:2270] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.085403 1170 raft_consensus.cc:357] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.086902 1170 raft_consensus.cc:383] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.087057 1170 raft_consensus.cc:738] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.087359 1170 consensus_queue.cc:260] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.088002 1170 ts_tablet_manager.cc:1428] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 13:51:52.088840 1170 tablet_bootstrap.cc:492] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:52.089362 1167 ts_tablet_manager.cc:1905] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.089510 1171 tablet_bootstrap.cc:492] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.089711 1171 ts_tablet_manager.cc:1397] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.122s user 0.035s sys 0.059s
I20250625 13:51:52.091178 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c87280e6a3e84e6ba5b0a90e1e7ecec3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:52.091629 1049 raft_consensus.cc:2466] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:52.092025 1166 tablet_bootstrap.cc:492] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.092131 1171 raft_consensus.cc:357] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.092234 1166 ts_tablet_manager.cc:1397] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20250625 13:51:52.092389 711 leader_election.cc:304] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters: affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.092983 1184 raft_consensus.cc:2802] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:52.093130 1184 raft_consensus.cc:491] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:52.093250 1184 raft_consensus.cc:3058] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.095736 1171 raft_consensus.cc:383] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.095842 1171 raft_consensus.cc:738] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.096074 1171 consensus_queue.cc:260] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.096143 1184 raft_consensus.cc:513] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.096613 1171 ts_tablet_manager.cc:1428] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250625 13:51:52.096776 1184 leader_election.cc:290] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.097190 1170 tablet_bootstrap.cc:654] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:52.097671 711 leader_election.cc:343] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: d455a2dec4304cf48ce29ceeb0700b0f
I20250625 13:51:52.097862 711 leader_election.cc:304] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.098234 1171 tablet_bootstrap.cc:492] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.098713 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c87280e6a3e84e6ba5b0a90e1e7ecec3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:52.099221 1184 raft_consensus.cc:2747] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.099565 1174 raft_consensus.cc:491] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.099745 1174 raft_consensus.cc:513] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.100044 1171 tablet_bootstrap.cc:654] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.100348 1174 leader_election.cc:290] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:52.101322 711 leader_election.cc:343] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.101773 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c87280e6a3e84e6ba5b0a90e1e7ecec3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:52.101840 1025 tablet_service.cc:1468] Processing CreateTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 (DEFAULT_TABLE table=default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.102015 1050 raft_consensus.cc:3058] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.102350 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5dda8b27b34dafb20d8ed8afc58460. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.103611 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6d92711284e64f71b31666800c250eab (table default.Table_fe20bac443bb450183cc5a10cdffff4c [id=96be558f5b694f3483d2501659dc5097]). Expected version 2 got 0
I20250625 13:51:52.105185 1050 raft_consensus.cc:2466] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:52.101265 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d92711284e64f71b31666800c250eab" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:52.105839 778 raft_consensus.cc:2466] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:52.093128 1166 raft_consensus.cc:357] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.105993 1166 raft_consensus.cc:383] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.106094 1166 raft_consensus.cc:738] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.106364 985 leader_election.cc:304] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:52.106297 1166 consensus_queue.cc:260] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.106981 1174 raft_consensus.cc:2802] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:52.107075 1174 raft_consensus.cc:491] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:52.107198 1174 raft_consensus.cc:3058] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.107877 1166 ts_tablet_manager.cc:1428] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250625 13:51:52.108191 1166 tablet_bootstrap.cc:492] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.108734 753 tablet_service.cc:1468] Processing CreateTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 (DEFAULT_TABLE table=default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.109155 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5dda8b27b34dafb20d8ed8afc58460. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.109827 1174 raft_consensus.cc:513] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.110154 1166 tablet_bootstrap.cc:654] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.111655 757 tablet_service.cc:1468] Processing CreateTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 (DEFAULT_TABLE table=default.Table_3d4e8fd3ed9144f2ba1aeb2cdea115e9 [id=e65eea160d8d473096f157b05be1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.112013 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4f9a46a5804da3aff1c9a87b2e4984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.115665 755 tablet_service.cc:1468] Processing CreateTablet for tablet 39dde16d7f144ef880b52abf9c979a53 (DEFAULT_TABLE table=default.Table_4e029acc913b4ce5b29a1fa12208e2cc [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.115952 758 tablet_service.cc:1468] Processing CreateTablet for tablet 7108932fa8a241819ca90674a64b285e (DEFAULT_TABLE table=default.Table_194007e5294849f9a6734e39b20d4fff [id=41a6138bc94a4a2790635b4ef0757c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.116129 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dde16d7f144ef880b52abf9c979a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.116286 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7108932fa8a241819ca90674a64b285e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.119597 754 tablet_service.cc:1468] Processing CreateTablet for tablet a179c3698d424ec7b6d87bf65cb858cc (DEFAULT_TABLE table=default.Table_1a827c72123c47dda8cb83ee349de91c [id=2c4408e7c06545faadb7f7b49a846782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.119957 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a179c3698d424ec7b6d87bf65cb858cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.123819 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d92711284e64f71b31666800c250eab" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:52.124054 916 raft_consensus.cc:2466] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:52.124913 750 tablet_service.cc:1468] Processing CreateTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f (DEFAULT_TABLE table=default.Table_70ae19ea7742453484d9525b11f16752 [id=c73f632380944ac9b8c2c181642922ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.125269 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a6375e98a04f95aa2bbde6b9b0228f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.126288 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a179c3698d424ec7b6d87bf65cb858cc (DEFAULT_TABLE table=default.Table_1a827c72123c47dda8cb83ee349de91c [id=2c4408e7c06545faadb7f7b49a846782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.126514 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 15750915ff7746ffa66509184be55dc4 (DEFAULT_TABLE table=default.Table_ffbe3938117d49e0b9627a5c8f1d45f4 [id=a64872263d0d4aeaaac5a6df29cb73bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.126849 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a179c3698d424ec7b6d87bf65cb858cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.127068 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 39dde16d7f144ef880b52abf9c979a53 (DEFAULT_TABLE table=default.Table_4e029acc913b4ce5b29a1fa12208e2cc [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.127564 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 7108932fa8a241819ca90674a64b285e (DEFAULT_TABLE table=default.Table_194007e5294849f9a6734e39b20d4fff [id=41a6138bc94a4a2790635b4ef0757c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.127898 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7108932fa8a241819ca90674a64b285e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.130889 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f (DEFAULT_TABLE table=default.Table_70ae19ea7742453484d9525b11f16752 [id=c73f632380944ac9b8c2c181642922ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.131398 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a6375e98a04f95aa2bbde6b9b0228f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.132244 896 tablet_service.cc:1468] Processing CreateTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 (DEFAULT_TABLE table=default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.132642 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5dda8b27b34dafb20d8ed8afc58460. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.132851 889 tablet_service.cc:1468] Processing CreateTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 (DEFAULT_TABLE table=default.Table_3d4e8fd3ed9144f2ba1aeb2cdea115e9 [id=e65eea160d8d473096f157b05be1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.133215 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4f9a46a5804da3aff1c9a87b2e4984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.126289 1030 tablet_service.cc:1468] Processing CreateTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 (DEFAULT_TABLE table=default.Table_3d4e8fd3ed9144f2ba1aeb2cdea115e9 [id=e65eea160d8d473096f157b05be1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.135586 895 tablet_service.cc:1468] Processing CreateTablet for tablet 15750915ff7746ffa66509184be55dc4 (DEFAULT_TABLE table=default.Table_ffbe3938117d49e0b9627a5c8f1d45f4 [id=a64872263d0d4aeaaac5a6df29cb73bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.135815 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4f9a46a5804da3aff1c9a87b2e4984. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.135939 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15750915ff7746ffa66509184be55dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.139639 893 tablet_service.cc:1468] Processing CreateTablet for tablet 39dde16d7f144ef880b52abf9c979a53 (DEFAULT_TABLE table=default.Table_4e029acc913b4ce5b29a1fa12208e2cc [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.139993 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dde16d7f144ef880b52abf9c979a53. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:52.143579 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a179c3698d424ec7b6d87bf65cb858cc consensus state has no leader
I20250625 13:51:52.143601 885 tablet_service.cc:1468] Processing CreateTablet for tablet 7108932fa8a241819ca90674a64b285e (DEFAULT_TABLE table=default.Table_194007e5294849f9a6734e39b20d4fff [id=41a6138bc94a4a2790635b4ef0757c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.143941 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7108932fa8a241819ca90674a64b285e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.143916 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 969dc4a9b811496aa3d926a3ae3965ea from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.144299 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 872724cd95bf473cb40e5f8ed8b5bd55
I20250625 13:51:52.144948 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 96d491878b8343be93892aa594ca9da7 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.145859 590 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6d92711284e64f71b31666800c250eab (table default.Table_fe20bac443bb450183cc5a10cdffff4c [id=96be558f5b694f3483d2501659dc5097]). Expected version 2 got 0
W20250625 13:51:52.146492 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d92711284e64f71b31666800c250eab consensus state has no leader
I20250625 13:51:52.146684 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a17f862a8a584047a8f0220f2eced869
I20250625 13:51:52.146755 595 master_service.cc:948] ReplaceTablet: received request to replace tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:52.147118 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0723e91650174f5badd74f7756c5a54e consensus state has no leader
I20250625 13:51:52.147264 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 07458e047b88493e9e52423b4e85b091 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.147593 891 tablet_service.cc:1468] Processing CreateTablet for tablet a179c3698d424ec7b6d87bf65cb858cc (DEFAULT_TABLE table=default.Table_1a827c72123c47dda8cb83ee349de91c [id=2c4408e7c06545faadb7f7b49a846782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.147626 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.147982 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a179c3698d424ec7b6d87bf65cb858cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.149104 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ab41f8d02dff471ba272dbf7620b9c6f
W20250625 13:51:52.150221 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 872724cd95bf473cb40e5f8ed8b5bd55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 872724cd95bf473cb40e5f8ed8b5bd55 already in progress: creating tablet
I20250625 13:51:52.115694 752 tablet_service.cc:1468] Processing CreateTablet for tablet 15750915ff7746ffa66509184be55dc4 (DEFAULT_TABLE table=default.Table_ffbe3938117d49e0b9627a5c8f1d45f4 [id=a64872263d0d4aeaaac5a6df29cb73bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.151981 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15750915ff7746ffa66509184be55dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.152968 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 07458e047b88493e9e52423b4e85b091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.153286 747 tablet_service.cc:1515] Processing DeleteTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.153700 1165 tablet_replica.cc:331] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.153959 1165 raft_consensus.cc:2241] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.154222 1165 raft_consensus.cc:2270] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.156452 1165 ts_tablet_manager.cc:1905] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.156620 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15750915ff7746ffa66509184be55dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.159353 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a17f862a8a584047a8f0220f2eced869 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.159725 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dde16d7f144ef880b52abf9c979a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.166615 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.166900 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f with delete_type TABLET_DATA_DELETED (replaced by tablet a17f862a8a584047a8f0220f2eced869) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.167026 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.166906 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet a17f862a8a584047a8f0220f2eced869 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.167624 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a5afade7a5d1447a98934e4d614075d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0723e91650174f5badd74f7756c5a54e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.168855 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd6be619b79429b956aae4a1bf1589d with delete_type TABLET_DATA_DELETED (replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.174577 1174 leader_election.cc:290] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:52.177291 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.177963 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.178737 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c6d92fec1b3f4f368c408fc302731df0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.178956 886 tablet_service.cc:1468] Processing CreateTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f (DEFAULT_TABLE table=default.Table_70ae19ea7742453484d9525b11f16752 [id=c73f632380944ac9b8c2c181642922ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.179678 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_04cf5db85d58479da934b19bc000f24d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.180248 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04cf5db85d58479da934b19bc000f24d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.180352 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_775545ba4c0343e394a73dc9327889ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.180907 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_775545ba4c0343e394a73dc9327889ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.181828 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3e92af655091475fa26f0d3e41d55306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.182399 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e92af655091475fa26f0d3e41d55306 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.183496 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_abfcbd249e4a4189aba6f16b5a66a0e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.183974 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abfcbd249e4a4189aba6f16b5a66a0e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.185001 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c282591e90c40948563fda06453e4d4 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:52.185683 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6d92fec1b3f4f368c408fc302731df0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.191401 711 leader_election.cc:304] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters: affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.191799 1184 raft_consensus.cc:2802] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:51:52.194478 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 88647.7us
I20250625 13:51:52.201274 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a6375e98a04f95aa2bbde6b9b0228f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.202330 892 tablet_service.cc:1515] Processing DeleteTablet for tablet a17f862a8a584047a8f0220f2eced869 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.204834 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d92711284e64f71b31666800c250eab" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:52.205054 916 raft_consensus.cc:3058] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.208007 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b44c85b12fe64fc99fcf0a79a5909c6c
I20250625 13:51:52.208349 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.208544 894 tablet_service.cc:1515] Processing DeleteTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.208885 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_500c159da8204c06b317eb5032e0113d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.209458 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_500c159da8204c06b317eb5032e0113d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:52.211413 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a17f862a8a584047a8f0220f2eced869 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a17f862a8a584047a8f0220f2eced869 already in progress: creating tablet
W20250625 13:51:52.211683 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 872724cd95bf473cb40e5f8ed8b5bd55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 872724cd95bf473cb40e5f8ed8b5bd55 already in progress: creating tablet
I20250625 13:51:52.213559 1168 tablet_replica.cc:331] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:52.213738 1168 raft_consensus.cc:2241] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.215617 1168 raft_consensus.cc:2270] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.216302 1168 ts_tablet_manager.cc:1905] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.223977 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ac5c66505c43484ca1990febfc2a5bf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.224371 916 raft_consensus.cc:2466] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:51:52.224496 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac5c66505c43484ca1990febfc2a5bf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.225008 983 leader_election.cc:304] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:52.225483 1174 raft_consensus.cc:2802] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:52.225844 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e75519f3b879406ea7bb8f515825f274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.226289 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e75519f3b879406ea7bb8f515825f274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.226946 1174 raft_consensus.cc:695] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:52.227432 1174 consensus_queue.cc:237] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.228148 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33015
--webserver_interface=127.0.104.62
--webserver_port=44303
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 13:51:52.230472 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d92711284e64f71b31666800c250eab" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:52.232597 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_84902149ef0541b2956407332a512a7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.233148 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84902149ef0541b2956407332a512a7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.237637 778 raft_consensus.cc:3058] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.238075 896 tablet_service.cc:1515] Processing DeleteTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.241708 1165 ts_tablet_manager.cc:1918] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.241837 1165 log.cc:1199] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/07458e047b88493e9e52423b4e85b091
I20250625 13:51:52.242542 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1fe0d8f47f6240cca9142f4fda013f5c" } new_table_name: "default.Table_cc3624b43cb14dfca4a12946b7198092" modify_external_catalogs: true
I20250625 13:51:52.249868 1165 ts_tablet_manager.cc:1939] T 07458e047b88493e9e52423b4e85b091 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.249910 753 tablet_service.cc:1515] Processing DeleteTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.250636 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_22c0e04fd41c41d981e66e94d5456772" } modify_external_catalogs: true
I20250625 13:51:52.250880 1165 tablet_replica.cc:331] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.250846 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_22c0e04fd41c41d981e66e94d5456772" } modify_external_catalogs: true
I20250625 13:51:52.251436 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 07458e047b88493e9e52423b4e85b091 (table default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd [id=aa9bb127a9054fe18cee9dded1f88cdf]) successfully deleted
I20250625 13:51:52.255366 1165 raft_consensus.cc:2241] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.255705 1165 raft_consensus.cc:2270] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.256819 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.257980 1168 ts_tablet_manager.cc:1918] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.258159 1168 log.cc:1199] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ab41f8d02dff471ba272dbf7620b9c6f
I20250625 13:51:52.258666 1168 ts_tablet_manager.cc:1939] T ab41f8d02dff471ba272dbf7620b9c6f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:52.260325 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet ab41f8d02dff471ba272dbf7620b9c6f (table default.Table_4c9213ada50e4b4089e8d20f31fd18e9 [id=d47dd54b4d984cc1874992118033256d]) successfully deleted
I20250625 13:51:52.261540 1165 ts_tablet_manager.cc:1905] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.262564 1220 raft_consensus.cc:491] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.262738 1220 raft_consensus.cc:513] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.264461 1167 ts_tablet_manager.cc:1918] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.264945 1167 log.cc:1199] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/20026a988f644f559ef967bcf9b46ed7
I20250625 13:51:52.265681 778 raft_consensus.cc:2466] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:52.266570 1220 leader_election.cc:290] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:51:52.267900 711 leader_election.cc:343] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: ecd6be619b79429b956aae4a1bf1589d
W20250625 13:51:52.271207 711 leader_election.cc:343] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: ecd6be619b79429b956aae4a1bf1589d
W20250625 13:51:52.272233 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet b44c85b12fe64fc99fcf0a79a5909c6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b44c85b12fe64fc99fcf0a79a5909c6c already in progress: creating tablet
I20250625 13:51:52.276217 1167 ts_tablet_manager.cc:1939] T 20026a988f644f559ef967bcf9b46ed7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.276531 711 leader_election.cc:304] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.276933 1220 raft_consensus.cc:2747] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.278049 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 20026a988f644f559ef967bcf9b46ed7 (table default.Table_a13af807e80443588d6008d9ce3d727c [id=e88514d0a01b411ba21f2fd2b4d615e5]) successfully deleted
I20250625 13:51:52.283245 1167 tablet_replica.cc:331] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.283535 1167 raft_consensus.cc:2241] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.283787 1167 raft_consensus.cc:2270] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.284786 1184 raft_consensus.cc:695] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:52.285504 1184 consensus_queue.cc:237] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.287223 1178 raft_consensus.cc:491] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.287423 1178 raft_consensus.cc:513] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.289532 1178 leader_election.cc:290] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.292439 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_775545ba4c0343e394a73dc9327889ff" } new_table_name: "default.Table_a9b04cfcee6646909171bdaf821ff85e" modify_external_catalogs: true
I20250625 13:51:52.299989 1167 ts_tablet_manager.cc:1905] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.299916 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "969dc4a9b811496aa3d926a3ae3965ea" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:52.300201 1050 raft_consensus.cc:2466] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:52.284835 1165 ts_tablet_manager.cc:1918] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.301153 849 leader_election.cc:304] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:52.301299 1165 log.cc:1199] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ab41f8d02dff471ba272dbf7620b9c6f
I20250625 13:51:52.301690 1165 ts_tablet_manager.cc:1939] T ab41f8d02dff471ba272dbf7620b9c6f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.301743 1178 raft_consensus.cc:2802] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:52.301882 1178 raft_consensus.cc:491] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:52.301995 1178 raft_consensus.cc:3058] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.304651 1021 tablet_service.cc:1468] Processing CreateTablet for tablet bf54376db991424b8de0e8f252fcb43f (DEFAULT_TABLE table=default.Table_9cc4881bfac74cfab175b6711d7b1f83 [id=8504f82376b440a9a3ab8c9bf1741cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.306383 1178 raft_consensus.cc:513] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.307749 1178 leader_election.cc:290] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.306980 886 tablet_service.cc:1468] Processing CreateTablet for tablet bf54376db991424b8de0e8f252fcb43f (DEFAULT_TABLE table=default.Table_9cc4881bfac74cfab175b6711d7b1f83 [id=8504f82376b440a9a3ab8c9bf1741cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.308343 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf54376db991424b8de0e8f252fcb43f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:52.315644 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76f074156559406c875e65796e6b5ee6 consensus state has no leader
I20250625 13:51:52.316159 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ab41f8d02dff471ba272dbf7620b9c6f (table default.Table_4c9213ada50e4b4089e8d20f31fd18e9 [id=d47dd54b4d984cc1874992118033256d]) successfully deleted
W20250625 13:51:52.316800 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a17f862a8a584047a8f0220f2eced869 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a17f862a8a584047a8f0220f2eced869 already in progress: creating tablet
W20250625 13:51:52.317101 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc has no consensus state
I20250625 13:51:52.308187 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "969dc4a9b811496aa3d926a3ae3965ea" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:52.318441 1050 raft_consensus.cc:3058] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:52.318857 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet b44c85b12fe64fc99fcf0a79a5909c6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b44c85b12fe64fc99fcf0a79a5909c6c already in progress: creating tablet
I20250625 13:51:52.319696 755 tablet_service.cc:1468] Processing CreateTablet for tablet bf54376db991424b8de0e8f252fcb43f (DEFAULT_TABLE table=default.Table_9cc4881bfac74cfab175b6711d7b1f83 [id=8504f82376b440a9a3ab8c9bf1741cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.320158 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf54376db991424b8de0e8f252fcb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.321177 1167 ts_tablet_manager.cc:1918] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.321329 1167 log.cc:1199] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/c87280e6a3e84e6ba5b0a90e1e7ecec3
I20250625 13:51:52.321362 1050 raft_consensus.cc:2466] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:52.321719 1167 ts_tablet_manager.cc:1939] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.321919 849 leader_election.cc:304] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:52.322576 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf54376db991424b8de0e8f252fcb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.322688 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 (table default.Table_d837899f74444b0a8e38179bb8dea2ee [id=bda1dd723374447b8a351345d7d3ae43]) successfully deleted
I20250625 13:51:52.327435 1178 raft_consensus.cc:2802] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:51:52.328550 851 leader_election.cc:343] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 969dc4a9b811496aa3d926a3ae3965ea
W20250625 13:51:52.329319 851 leader_election.cc:343] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 969dc4a9b811496aa3d926a3ae3965ea
I20250625 13:51:52.334445 1170 tablet_bootstrap.cc:492] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:52.334704 1170 ts_tablet_manager.cc:1397] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.246s user 0.028s sys 0.085s
I20250625 13:51:52.335588 1170 raft_consensus.cc:357] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.335806 1170 raft_consensus.cc:383] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.335901 1170 raft_consensus.cc:738] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.336174 1170 consensus_queue.cc:260] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.336999 1171 tablet_bootstrap.cc:492] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.337199 1171 ts_tablet_manager.cc:1397] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.239s user 0.007s sys 0.015s
I20250625 13:51:52.337937 1171 raft_consensus.cc:357] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.338280 1171 raft_consensus.cc:383] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.338377 1171 raft_consensus.cc:738] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.338611 1170 ts_tablet_manager.cc:1428] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 13:51:52.338637 1171 consensus_queue.cc:260] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.339169 1171 ts_tablet_manager.cc:1428] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250625 13:51:52.339414 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_4afe38f204684075b394ad74549a04c2 [id=c2e58d3856d7401ca7b97d9cce92bbce], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0723e91650174f5badd74f7756c5a54e consensus state has no leader
I20250625 13:51:52.339681 1171 tablet_bootstrap.cc:492] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.341368 1171 tablet_bootstrap.cc:654] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.342168 1166 tablet_bootstrap.cc:492] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.342759 1166 ts_tablet_manager.cc:1397] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.235s user 0.036s sys 0.061s
I20250625 13:51:52.343696 1166 raft_consensus.cc:357] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.344103 1166 raft_consensus.cc:383] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.344388 1166 raft_consensus.cc:738] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.344810 1166 consensus_queue.cc:260] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.345492 1166 ts_tablet_manager.cc:1428] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:52.345854 1166 tablet_bootstrap.cc:492] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.345927 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3c282591e90c40948563fda06453e4d4
I20250625 13:51:52.346393 596 catalog_manager.cc:6652] ReplaceTablet: tablet 3c282591e90c40948563fda06453e4d4 deleted and replaced by tablet 60d6dcfd452448ecb242d863ee277170
I20250625 13:51:52.347353 1170 tablet_bootstrap.cc:492] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:52.348632 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c282591e90c40948563fda06453e4d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 60d6dcfd452448ecb242d863ee277170) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.349090 1170 tablet_bootstrap.cc:654] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.349076 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4e029acc913b4ce5b29a1fa12208e2cc" } new_table_name: "default.Table_489406ccd1ad4241b50f9b8d937df5de" modify_external_catalogs: true
I20250625 13:51:52.350190 1167 tablet_replica.cc:331] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.350374 1167 raft_consensus.cc:2241] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:52.350418 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3c282591e90c40948563fda06453e4d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c282591e90c40948563fda06453e4d4 already in progress: creating tablet
I20250625 13:51:52.350559 1167 raft_consensus.cc:2270] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.351263 1166 tablet_bootstrap.cc:654] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.353751 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9cc4881bfac74cfab175b6711d7b1f83" } new_table_name: "default.Table_719d67c948c8495ba1f5c2001c020ce5" modify_external_catalogs: true
I20250625 13:51:52.355477 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e92af655091475fa26f0d3e41d55306" } new_table_name: "default.Table_aadd4b5489ea4e79b31a6d801a46cc87" modify_external_catalogs: true
I20250625 13:51:52.356884 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_abfcbd249e4a4189aba6f16b5a66a0e7" } modify_external_catalogs: true
I20250625 13:51:52.357077 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_abfcbd249e4a4189aba6f16b5a66a0e7" } modify_external_catalogs: true
I20250625 13:51:52.358587 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ac5c66505c43484ca1990febfc2a5bf9" } modify_external_catalogs: true
I20250625 13:51:52.358753 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ac5c66505c43484ca1990febfc2a5bf9" } modify_external_catalogs: true
I20250625 13:51:52.358960 1170 tablet_bootstrap.cc:492] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:52.359107 1170 ts_tablet_manager.cc:1397] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20250625 13:51:52.359897 1167 ts_tablet_manager.cc:1905] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.360801 1171 tablet_bootstrap.cc:492] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.360023 1170 raft_consensus.cc:357] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.360968 1171 ts_tablet_manager.cc:1397] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250625 13:51:52.361013 1170 raft_consensus.cc:383] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.361204 1170 raft_consensus.cc:738] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.361446 1166 tablet_bootstrap.cc:492] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.361605 1166 ts_tablet_manager.cc:1397] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20250625 13:51:52.361492 1170 consensus_queue.cc:260] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.361850 1171 raft_consensus.cc:357] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.361893 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e" } modify_external_catalogs: true
I20250625 13:51:52.362033 1171 raft_consensus.cc:383] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.362040 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e" } modify_external_catalogs: true
I20250625 13:51:52.362126 1171 raft_consensus.cc:738] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.362344 1170 ts_tablet_manager.cc:1428] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250625 13:51:52.362955 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c282591e90c40948563fda06453e4d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 60d6dcfd452448ecb242d863ee277170) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.363476 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c282591e90c40948563fda06453e4d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 60d6dcfd452448ecb242d863ee277170) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:52.363745 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3c282591e90c40948563fda06453e4d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c282591e90c40948563fda06453e4d4 already in progress: creating tablet
I20250625 13:51:52.364580 1166 raft_consensus.cc:357] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.364856 1166 raft_consensus.cc:383] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.365025 1166 raft_consensus.cc:738] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.365345 1166 consensus_queue.cc:260] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.366040 1166 ts_tablet_manager.cc:1428] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:51:52.366369 1166 tablet_bootstrap.cc:492] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.365634 1171 consensus_queue.cc:260] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:52.367831 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39dde16d7f144ef880b52abf9c979a53 consensus state has no leader
I20250625 13:51:52.369416 1171 ts_tablet_manager.cc:1428] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:51:52.369700 1170 tablet_bootstrap.cc:492] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:52.370389 1171 tablet_bootstrap.cc:492] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.371488 1170 tablet_bootstrap.cc:654] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.371940 1171 tablet_bootstrap.cc:654] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.373660 1166 tablet_bootstrap.cc:654] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.374928 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6d92711284e64f71b31666800c250eab (table default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097]). Expected version 3 got 0
I20250625 13:51:52.376291 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6d92711284e64f71b31666800c250eab (table default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097]). Expected version 3 got 0
I20250625 13:51:52.379155 757 tablet_service.cc:1515] Processing DeleteTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.379803 750 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd6be619b79429b956aae4a1bf1589d with delete_type TABLET_DATA_DELETED (replaced by tablet b15fb962c53f46f890d8f11cae6eb4a2) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:52.379935 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ab41f8d02dff471ba272dbf7620b9c6f because the tablet was not found. No further retry: Not found: Tablet not found: ab41f8d02dff471ba272dbf7620b9c6f
I20250625 13:51:52.380065 1165 tablet_replica.cc:331] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.380245 1165 raft_consensus.cc:2241] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.380303 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 07458e047b88493e9e52423b4e85b091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.380430 1165 raft_consensus.cc:2270] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.380617 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ab41f8d02dff471ba272dbf7620b9c6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.380815 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.381008 1168 tablet_replica.cc:331] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:52.381191 1168 raft_consensus.cc:2241] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.381348 1168 raft_consensus.cc:2270] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.381420 1165 ts_tablet_manager.cc:1905] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.382270 1168 ts_tablet_manager.cc:1905] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.384268 758 tablet_service.cc:1515] Processing DeleteTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:52.384630 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce84f74470ad428abb2113efa170ca24 has no consensus state
I20250625 13:51:52.384934 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 09a8e15d57f14702ba0b5b52a94fc7c4
W20250625 13:51:52.385885 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf54376db991424b8de0e8f252fcb43f consensus state has no leader
I20250625 13:51:52.387919 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.388388 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.389206 755 tablet_service.cc:1515] Processing DeleteTablet for tablet d455a2dec4304cf48ce29ceeb0700b0f with delete_type TABLET_DATA_DELETED (replaced by tablet a17f862a8a584047a8f0220f2eced869) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.389483 752 tablet_service.cc:1515] Processing DeleteTablet for tablet a5afade7a5d1447a98934e4d614075d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0723e91650174f5badd74f7756c5a54e) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.390309 1168 ts_tablet_manager.cc:1918] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.390455 1168 log.cc:1199] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/07458e047b88493e9e52423b4e85b091
I20250625 13:51:52.390658 1174 raft_consensus.cc:491] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.390818 1174 raft_consensus.cc:513] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.390931 1168 ts_tablet_manager.cc:1939] T 07458e047b88493e9e52423b4e85b091 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:52.391032 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.391530 1174 leader_election.cc:290] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:52.391983 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 07458e047b88493e9e52423b4e85b091 (table default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd [id=aa9bb127a9054fe18cee9dded1f88cdf]) successfully deleted
I20250625 13:51:52.392513 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92a90b0dc85844a892ab247ae0d09a12" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:52.392637 581 catalog_manager.cc:4895] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ab41f8d02dff471ba272dbf7620b9c6f because the tablet was not found. No further retry: Not found: Tablet not found: ab41f8d02dff471ba272dbf7620b9c6f
I20250625 13:51:52.392964 1168 tablet_replica.cc:331] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:52.393013 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92a90b0dc85844a892ab247ae0d09a12" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:52.393600 983 leader_election.cc:343] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.393769 1167 ts_tablet_manager.cc:1918] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.393908 1167 log.cc:1199] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/969dc4a9b811496aa3d926a3ae3965ea
I20250625 13:51:52.394279 1167 ts_tablet_manager.cc:1939] T 969dc4a9b811496aa3d926a3ae3965ea P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.395484 754 tablet_service.cc:1515] Processing DeleteTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:52.397069 985 leader_election.cc:343] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.397256 985 leader_election.cc:304] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.397797 1174 raft_consensus.cc:2747] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.400092 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 969dc4a9b811496aa3d926a3ae3965ea (table default.Table_d64bbacfddeb4945b2396e41e59cbb23 [id=f00b1eb97d6245a8b22ad21216ed5230]) successfully deleted
I20250625 13:51:52.400907 1167 tablet_replica.cc:331] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.401608 1167 raft_consensus.cc:2241] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.402204 1165 ts_tablet_manager.cc:1918] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.402381 1165 log.cc:1199] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ecd6be619b79429b956aae4a1bf1589d
I20250625 13:51:52.402292 1167 raft_consensus.cc:2270] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.402829 1165 ts_tablet_manager.cc:1939] T ecd6be619b79429b956aae4a1bf1589d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.403591 1167 ts_tablet_manager.cc:1905] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.405180 1171 tablet_bootstrap.cc:492] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.405369 1171 ts_tablet_manager.cc:1397] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.007s
I20250625 13:51:52.405972 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 15750915ff7746ffa66509184be55dc4 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.406199 1171 raft_consensus.cc:357] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.406411 1171 raft_consensus.cc:383] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.406474 1171 raft_consensus.cc:738] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.406713 1171 consensus_queue.cc:260] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.407122 590 master_service.cc:948] ReplaceTablet: received request to replace tablet a418d2a6c8f24043a820967489c87e23 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.408164 588 master_service.cc:948] ReplaceTablet: received request to replace tablet ca5dda8b27b34dafb20d8ed8afc58460 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.408392 1171 ts_tablet_manager.cc:1428] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:51:52.409502 1171 tablet_bootstrap.cc:492] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.409770 594 catalog_manager.cc:5582] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3). New cstate: current_term: 1 leader_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: HEALTHY } } }
I20250625 13:51:52.410725 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 872724cd95bf473cb40e5f8ed8b5bd55 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.410908 594 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6d92711284e64f71b31666800c250eab (table default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097]). Expected version 3 got 0
I20250625 13:51:52.411378 1171 tablet_bootstrap.cc:654] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.406076 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 969dc4a9b811496aa3d926a3ae3965ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.406226 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 20026a988f644f559ef967bcf9b46ed7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.406358 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.416241 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e75519f3b879406ea7bb8f515825f274" } modify_external_catalogs: true
I20250625 13:51:52.416462 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e75519f3b879406ea7bb8f515825f274" } modify_external_catalogs: true
I20250625 13:51:52.418205 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b4b212f7c1ab48d2807a9eebeec31c07
I20250625 13:51:52.419250 1171 tablet_bootstrap.cc:492] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.419469 1171 ts_tablet_manager.cc:1397] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250625 13:51:52.419498 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b4b212f7c1ab48d2807a9eebeec31c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.419811 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b4b212f7c1ab48d2807a9eebeec31c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.420328 1171 raft_consensus.cc:357] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.420490 1171 raft_consensus.cc:383] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.420588 1171 raft_consensus.cc:738] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.420811 1171 consensus_queue.cc:260] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.421306 754 tablet_service.cc:1515] Processing DeleteTablet for tablet b4b212f7c1ab48d2807a9eebeec31c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.422683 1171 ts_tablet_manager.cc:1428] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:51:52.423908 1171 tablet_bootstrap.cc:492] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.424355 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7a3fe0cc014e4cfd9e21d44e013bfc8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.425055 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a3fe0cc014e4cfd9e21d44e013bfc8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.425794 1171 tablet_bootstrap.cc:654] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.428087 1171 tablet_bootstrap.cc:492] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.428234 1171 ts_tablet_manager.cc:1397] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20250625 13:51:52.428939 1171 raft_consensus.cc:357] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.429124 1171 raft_consensus.cc:383] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.429183 1171 raft_consensus.cc:738] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.429491 1171 consensus_queue.cc:260] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.430006 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 15750915ff7746ffa66509184be55dc4
I20250625 13:51:52.430552 595 catalog_manager.cc:6652] ReplaceTablet: tablet 15750915ff7746ffa66509184be55dc4 deleted and replaced by tablet 6e0f2fa5d5614ad394e6e12ec62814d4
I20250625 13:51:52.430898 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a418d2a6c8f24043a820967489c87e23
I20250625 13:51:52.431175 1171 ts_tablet_manager.cc:1428] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:51:52.431210 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 15750915ff7746ffa66509184be55dc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e0f2fa5d5614ad394e6e12ec62814d4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.431386 590 catalog_manager.cc:6652] ReplaceTablet: tablet a418d2a6c8f24043a820967489c87e23 deleted and replaced by tablet dcb392fc4235435db39b0714b0270ee9
I20250625 13:51:52.431521 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 15750915ff7746ffa66509184be55dc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e0f2fa5d5614ad394e6e12ec62814d4) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.431784 758 tablet_service.cc:1515] Processing DeleteTablet for tablet a418d2a6c8f24043a820967489c87e23 with delete_type TABLET_DATA_DELETED (replaced by tablet dcb392fc4235435db39b0714b0270ee9) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.432138 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a418d2a6c8f24043a820967489c87e23 with delete_type TABLET_DATA_DELETED (replaced by tablet dcb392fc4235435db39b0714b0270ee9) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.432439 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a418d2a6c8f24043a820967489c87e23 with delete_type TABLET_DATA_DELETED (replaced by tablet dcb392fc4235435db39b0714b0270ee9) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.432586 1171 tablet_bootstrap.cc:492] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.434489 1171 tablet_bootstrap.cc:654] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.437630 1167 ts_tablet_manager.cc:1918] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.437844 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 15750915ff7746ffa66509184be55dc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e0f2fa5d5614ad394e6e12ec62814d4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.441192 1167 log.cc:1199] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/07458e047b88493e9e52423b4e85b091
I20250625 13:51:52.442112 1167 ts_tablet_manager.cc:1939] T 07458e047b88493e9e52423b4e85b091 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.441246 1178 raft_consensus.cc:695] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:52.444579 1178 consensus_queue.cc:237] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.445397 1174 raft_consensus.cc:491] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.445581 1174 raft_consensus.cc:513] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.446434 1174 leader_election.cc:290] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:52.447285 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dbeb984fadf4a0ebc50087c1a921f07" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:52.447867 1167 tablet_replica.cc:331] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.448339 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dbeb984fadf4a0ebc50087c1a921f07" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:52.448315 985 leader_election.cc:343] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.448704 1167 raft_consensus.cc:2241] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:52.448843 983 leader_election.cc:343] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.449321 983 leader_election.cc:304] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.449810 1174 raft_consensus.cc:2747] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.450271 1167 raft_consensus.cc:2270] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.451629 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fdde488d5ee94a8593bf9fc6205adeac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.452428 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdde488d5ee94a8593bf9fc6205adeac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.452494 1167 ts_tablet_manager.cc:1905] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.456374 1171 tablet_bootstrap.cc:492] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.456535 1171 ts_tablet_manager.cc:1397] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.011s
I20250625 13:51:52.457587 1171 raft_consensus.cc:357] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.457823 1171 raft_consensus.cc:383] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.457926 1171 raft_consensus.cc:738] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.458212 1171 consensus_queue.cc:260] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.460455 1171 ts_tablet_manager.cc:1428] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250625 13:51:52.461341 1227 raft_consensus.cc:491] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.461679 1171 tablet_bootstrap.cc:492] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.461536 1227 raft_consensus.cc:513] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.462483 1227 leader_election.cc:290] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.462940 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b44c85b12fe64fc99fcf0a79a5909c6c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:52.463564 851 leader_election.cc:343] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.464257 1171 tablet_bootstrap.cc:654] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.465968 1219 raft_consensus.cc:491] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.466277 656 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:52.466589 1219 raft_consensus.cc:513] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.468115 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.469409 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b44c85b12fe64fc99fcf0a79a5909c6c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:52.470000 849 leader_election.cc:343] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.470144 849 leader_election.cc:304] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:52.470679 1227 raft_consensus.cc:2747] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.471786 1219 leader_election.cc:290] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.472292 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7906d7c2fb541dcb1ec8e8dcc729c25" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:52.472505 916 raft_consensus.cc:2466] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:52.473874 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7906d7c2fb541dcb1ec8e8dcc729c25" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:52.475189 1050 raft_consensus.cc:2466] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:52.475698 1171 tablet_bootstrap.cc:492] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.475934 1171 ts_tablet_manager.cc:1397] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20250625 13:51:52.476833 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7fdbbd3847ab4eb6b3f9872d5b0d9c4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.476976 1171 raft_consensus.cc:357] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.477147 1171 raft_consensus.cc:383] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.477210 1171 raft_consensus.cc:738] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:51:52.477478 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fdbbd3847ab4eb6b3f9872d5b0d9c4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.477460 1171 consensus_queue.cc:260] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.478016 1171 ts_tablet_manager.cc:1428] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:52.478129 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ca5dda8b27b34dafb20d8ed8afc58460
I20250625 13:51:52.478524 1171 tablet_bootstrap.cc:492] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.478694 588 catalog_manager.cc:6652] ReplaceTablet: tablet ca5dda8b27b34dafb20d8ed8afc58460 deleted and replaced by tablet 4b549bc8cead4af4a72f41f51655602c
I20250625 13:51:52.480526 1171 tablet_bootstrap.cc:654] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.481717 1167 ts_tablet_manager.cc:1918] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.481891 1167 log.cc:1199] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ab41f8d02dff471ba272dbf7620b9c6f
I20250625 13:51:52.481875 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_18f2b0c4409b45848a64bb65f651c59e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.481946 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fb2e294ac01849e6bffc287498f9dcb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.482246 1167 ts_tablet_manager.cc:1939] T ab41f8d02dff471ba272dbf7620b9c6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:51:52.482496 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2e294ac01849e6bffc287498f9dcb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.483603 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d9dd2f12f8bb4f6cba435c892dac832e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.484051 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9dd2f12f8bb4f6cba435c892dac832e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.484874 1167 tablet_replica.cc:331] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:51:52.485237 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18f2b0c4409b45848a64bb65f651c59e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.485827 1167 raft_consensus.cc:2241] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.486011 1167 raft_consensus.cc:2270] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.487187 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ecd6be619b79429b956aae4a1bf1589d (table default.Table_f0748f143cf540dbbe1d769b411ac975 [id=d298fec0d0c14fc297e4a3b4f82b007a]) successfully deleted
I20250625 13:51:52.487768 1167 ts_tablet_manager.cc:1905] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.473069 711 leader_election.cc:304] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:52.489686 1220 raft_consensus.cc:2802] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:52.473778 754 tablet_service.cc:1468] Processing CreateTablet for tablet b11b3157dd764f229831c3b8417460c6 (DEFAULT_TABLE table=default.Table_84902149ef0541b2956407332a512a7a [id=75b34bf006e040d18266d0c90986afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.483289 758 tablet_service.cc:1515] Processing DeleteTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b549bc8cead4af4a72f41f51655602c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.489794 1220 raft_consensus.cc:491] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:52.489915 1220 raft_consensus.cc:3058] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.490166 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11b3157dd764f229831c3b8417460c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.492996 1220 raft_consensus.cc:513] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.489166 1165 tablet_replica.cc:331] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.494063 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7906d7c2fb541dcb1ec8e8dcc729c25" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:52.494283 916 raft_consensus.cc:3058] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.494356 1165 raft_consensus.cc:2241] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.494510 1165 raft_consensus.cc:2270] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.495080 1171 tablet_bootstrap.cc:492] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.495216 1171 ts_tablet_manager.cc:1397] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.006s
I20250625 13:51:52.496043 1171 raft_consensus.cc:357] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.496209 1171 raft_consensus.cc:383] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.496140 758 tablet_service.cc:1468] Processing CreateTablet for tablet e75b5fb51373451a94cb5ca59cdcefac (DEFAULT_TABLE table=default.Table_e75519f3b879406ea7bb8f515825f274 [id=ffb244ee24124e8a83db14b5747abed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.496294 1171 raft_consensus.cc:738] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.496541 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75b5fb51373451a94cb5ca59cdcefac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.496497 1171 consensus_queue.cc:260] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.497004 1171 ts_tablet_manager.cc:1428] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:51:52.497212 916 raft_consensus.cc:2466] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:52.497277 1171 tablet_bootstrap.cc:492] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.497962 711 leader_election.cc:304] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:51:52.498180 711 raft_consensus.cc:2684] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:51:52.498631 1171 tablet_bootstrap.cc:654] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.498751 1220 leader_election.cc:290] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.499226 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7906d7c2fb541dcb1ec8e8dcc729c25" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:52.499454 1050 raft_consensus.cc:3058] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.503209 1050 raft_consensus.cc:2466] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:52.509699 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3599147b4b6b4aa3bcaa6de4475ae800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.510399 1167 ts_tablet_manager.cc:1918] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.510555 1167 log.cc:1199] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a17f862a8a584047a8f0220f2eced869
W20250625 13:51:52.510471 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3599147b4b6b4aa3bcaa6de4475ae800 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.510946 1167 ts_tablet_manager.cc:1939] T a17f862a8a584047a8f0220f2eced869 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.510795 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6199817767f74031b8a500d21174d5a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.512658 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6199817767f74031b8a500d21174d5a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.514358 1167 tablet_replica.cc:331] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.514565 1167 raft_consensus.cc:2241] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.514765 1167 raft_consensus.cc:2270] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.515439 1167 ts_tablet_manager.cc:1905] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.520080 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d
I20250625 13:51:52.522100 1167 ts_tablet_manager.cc:1918] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.522229 1167 log.cc:1199] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/e7906d7c2fb541dcb1ec8e8dcc729c25
I20250625 13:51:52.522652 1167 ts_tablet_manager.cc:1939] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.524142 758 tablet_service.cc:1515] Processing DeleteTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.524982 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6e5d1377cfc84651afd53a78735468d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.525743 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e5d1377cfc84651afd53a78735468d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.529218 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 07458e047b88493e9e52423b4e85b091 (table default.Table_2a5b3f1ce4b44df28a373af42d2ef5bd [id=aa9bb127a9054fe18cee9dded1f88cdf]) successfully deleted
W20250625 13:51:52.529467 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 872724cd95bf473cb40e5f8ed8b5bd55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 872724cd95bf473cb40e5f8ed8b5bd55 already in progress: creating tablet
W20250625 13:51:52.529701 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d455a2dec4304cf48ce29ceeb0700b0f because the tablet was not found. No further retry: Not found: Tablet not found: d455a2dec4304cf48ce29ceeb0700b0f
I20250625 13:51:52.529874 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ab41f8d02dff471ba272dbf7620b9c6f (table default.Table_4c9213ada50e4b4089e8d20f31fd18e9 [id=d47dd54b4d984cc1874992118033256d]) successfully deleted
I20250625 13:51:52.530052 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a17f862a8a584047a8f0220f2eced869 (table default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]) successfully deleted
W20250625 13:51:52.530272 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a5afade7a5d1447a98934e4d614075d8 because the tablet was not found. No further retry: Not found: Tablet not found: a5afade7a5d1447a98934e4d614075d8
I20250625 13:51:52.530251 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:51:52.530460 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ecd6be619b79429b956aae4a1bf1589d because the tablet was not found. No further retry: Not found: Tablet not found: ecd6be619b79429b956aae4a1bf1589d
W20250625 13:51:52.530639 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b44c85b12fe64fc99fcf0a79a5909c6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b44c85b12fe64fc99fcf0a79a5909c6c already in progress: creating tablet
I20250625 13:51:52.530695 893 tablet_service.cc:1468] Processing CreateTablet for tablet b11b3157dd764f229831c3b8417460c6 (DEFAULT_TABLE table=default.Table_84902149ef0541b2956407332a512a7a [id=75b34bf006e040d18266d0c90986afeb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:52.530840 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3c282591e90c40948563fda06453e4d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c282591e90c40948563fda06453e4d4 already in progress: creating tablet
W20250625 13:51:52.531023 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 because the tablet was not found. No further retry: Not found: Tablet not found: 09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:52.531195 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11b3157dd764f229831c3b8417460c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:52.531212 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 969dc4a9b811496aa3d926a3ae3965ea because the tablet was not found. No further retry: Not found: Tablet not found: 969dc4a9b811496aa3d926a3ae3965ea
W20250625 13:51:52.531410 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 20026a988f644f559ef967bcf9b46ed7 because the tablet was not found. No further retry: Not found: Tablet not found: 20026a988f644f559ef967bcf9b46ed7
I20250625 13:51:52.531579 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet e7906d7c2fb541dcb1ec8e8dcc729c25 (table default.Table_c32eb049fff54229a1db62dc389476aa [id=bb31d8b38dd4479694c08744f757ef7d]) successfully deleted
W20250625 13:51:52.531778 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b4b212f7c1ab48d2807a9eebeec31c07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4b212f7c1ab48d2807a9eebeec31c07 already in progress: creating tablet
W20250625 13:51:52.531961 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 15750915ff7746ffa66509184be55dc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15750915ff7746ffa66509184be55dc4 already in progress: creating tablet
W20250625 13:51:52.532119 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a418d2a6c8f24043a820967489c87e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a418d2a6c8f24043a820967489c87e23 already in progress: creating tablet
I20250625 13:51:52.534296 886 tablet_service.cc:1515] Processing DeleteTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b549bc8cead4af4a72f41f51655602c) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.534786 885 tablet_service.cc:1468] Processing CreateTablet for tablet e75b5fb51373451a94cb5ca59cdcefac (DEFAULT_TABLE table=default.Table_e75519f3b879406ea7bb8f515825f274 [id=ffb244ee24124e8a83db14b5747abed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.535195 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75b5fb51373451a94cb5ca59cdcefac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.538439 895 tablet_service.cc:1515] Processing DeleteTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.539052 1022 tablet_service.cc:1468] Processing CreateTablet for tablet b11b3157dd764f229831c3b8417460c6 (DEFAULT_TABLE table=default.Table_84902149ef0541b2956407332a512a7a [id=75b34bf006e040d18266d0c90986afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.539539 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11b3157dd764f229831c3b8417460c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.543494 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.543737 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b549bc8cead4af4a72f41f51655602c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.543643 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_969aea61c59149e39d51efed86647070"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.544091 1020 tablet_service.cc:1468] Processing CreateTablet for tablet e75b5fb51373451a94cb5ca59cdcefac (DEFAULT_TABLE table=default.Table_e75519f3b879406ea7bb8f515825f274 [id=ffb244ee24124e8a83db14b5747abed7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:52.544199 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_969aea61c59149e39d51efed86647070 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.544464 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75b5fb51373451a94cb5ca59cdcefac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.545410 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d0cf3d979369419882a2ba2083933d48
I20250625 13:51:52.546057 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_500c159da8204c06b317eb5032e0113d" } new_table_name: "default.Table_888aed1788b245e5b873ac73a0dcfba5" modify_external_catalogs: true
I20250625 13:51:52.548807 758 tablet_service.cc:1515] Processing DeleteTablet for tablet d0cf3d979369419882a2ba2083933d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.549288 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.556440 1227 raft_consensus.cc:491] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.556630 1227 raft_consensus.cc:513] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:52.557353 1227 leader_election.cc:290] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:52.557984 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c87280e6a3e84e6ba5b0a90e1e7ecec3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:52.558209 849 leader_election.cc:343] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: c87280e6a3e84e6ba5b0a90e1e7ecec3
I20250625 13:51:52.558836 851 leader_election.cc:304] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:52.559372 1227 raft_consensus.cc:2747] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.560400 1227 raft_consensus.cc:491] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.560528 1227 raft_consensus.cc:513] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.561153 1227 leader_election.cc:290] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.561751 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a418d2a6c8f24043a820967489c87e23" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:52.561849 1220 raft_consensus.cc:491] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.562008 1220 raft_consensus.cc:513] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:52.562292 849 leader_election.cc:343] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.562635 1220 leader_election.cc:290] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:51:52.563669 711 leader_election.cc:343] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: a5afade7a5d1447a98934e4d614075d8
I20250625 13:51:52.564157 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a418d2a6c8f24043a820967489c87e23" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:52.564714 851 leader_election.cc:343] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.564847 851 leader_election.cc:304] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:51:52.565672 711 leader_election.cc:343] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: a5afade7a5d1447a98934e4d614075d8
I20250625 13:51:52.565807 711 leader_election.cc:304] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.566272 1220 raft_consensus.cc:2747] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.566537 1227 raft_consensus.cc:2747] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.570756 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_04cf5db85d58479da934b19bc000f24d" } new_table_name: "default.Table_5b0408ce393341fb891ee6511dbf4890" modify_external_catalogs: true
I20250625 13:51:52.571720 1167 tablet_replica.cc:331] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.571933 1167 raft_consensus.cc:2241] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.572113 1167 raft_consensus.cc:2270] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.574610 1167 ts_tablet_manager.cc:1905] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.577104 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 3dbeb984fadf4a0ebc50087c1a921f07 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.581272 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3d4e8fd3ed9144f2ba1aeb2cdea115e9" } new_table_name: "default.Table_1481e562293146f799b1fbea03c28dc7" modify_external_catalogs: true
I20250625 13:51:52.583464 1165 ts_tablet_manager.cc:1905] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.585929 1168 raft_consensus.cc:2241] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:52.586910 1168 pending_rounds.cc:70] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:51:52.587095 1168 pending_rounds.cc:77] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520563144323072 op_type: NO_OP noop_request { }
I20250625 13:51:52.587288 1168 raft_consensus.cc:2887] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:52.587455 1168 raft_consensus.cc:2270] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.589988 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e75b5fb51373451a94cb5ca59cdcefac
I20250625 13:51:52.595506 1168 ts_tablet_manager.cc:1905] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.602813 1168 ts_tablet_manager.cc:1918] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:52.603098 1168 log.cc:1199] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/969dc4a9b811496aa3d926a3ae3965ea
I20250625 13:51:52.603550 1168 ts_tablet_manager.cc:1939] T 969dc4a9b811496aa3d926a3ae3965ea P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:52.605057 1168 tablet_replica.cc:331] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:52.605253 1168 raft_consensus.cc:2241] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.605448 1168 raft_consensus.cc:2270] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.593887 752 tablet_service.cc:1515] Processing DeleteTablet for tablet e75b5fb51373451a94cb5ca59cdcefac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.594033 757 tablet_service.cc:1468] Processing CreateTablet for tablet d0cf3d979369419882a2ba2083933d48 (DEFAULT_TABLE table=default.Table_ac5c66505c43484ca1990febfc2a5bf9 [id=eb254e00e07047a09c1924a6cab20e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.606495 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cf3d979369419882a2ba2083933d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.594233 755 tablet_service.cc:1468] Processing CreateTablet for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (DEFAULT_TABLE table=default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.611660 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.611912 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aadd4b5489ea4e79b31a6d801a46cc87" } new_table_name: "default.Table_2c63842cb66d43c19e91b079c23ad1df" modify_external_catalogs: true
I20250625 13:51:52.613425 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_489406ccd1ad4241b50f9b8d937df5de" } new_table_name: "default.Table_8013d5185a734552a6261f57235f3689" modify_external_catalogs: true
I20250625 13:51:52.594581 750 tablet_service.cc:1468] Processing CreateTablet for tablet ce84f74470ad428abb2113efa170ca24 (DEFAULT_TABLE table=default.Table_aadd4b5489ea4e79b31a6d801a46cc87 [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.615689 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce84f74470ad428abb2113efa170ca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.616178 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
I20250625 13:51:52.617304 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_893b774d7d574f8e89a027212ab17b74" } modify_external_catalogs: true
I20250625 13:51:52.617460 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_893b774d7d574f8e89a027212ab17b74" } modify_external_catalogs: true
I20250625 13:51:52.594947 753 tablet_service.cc:1468] Processing CreateTablet for tablet 32221a413957450fa59ffcfbc91c51b9 (DEFAULT_TABLE table=default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.619942 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32221a413957450fa59ffcfbc91c51b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:52.635056 581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:33015, user_credentials={real_user=slave}} blocked reactor thread for 101293us
I20250625 13:51:52.595956 758 tablet_service.cc:1468] Processing CreateTablet for tablet 99090fee41d44946b6243b1b4bff3635 (DEFAULT_TABLE table=default.Table_500c159da8204c06b317eb5032e0113d [id=c22a16ca0ca649238ac510cb5542bd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.636526 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99090fee41d44946b6243b1b4bff3635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.637243 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet d0cf3d979369419882a2ba2083933d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.638191 893 tablet_service.cc:1468] Processing CreateTablet for tablet 99090fee41d44946b6243b1b4bff3635 (DEFAULT_TABLE table=default.Table_500c159da8204c06b317eb5032e0113d [id=c22a16ca0ca649238ac510cb5542bd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.638660 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99090fee41d44946b6243b1b4bff3635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.638702 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 99090fee41d44946b6243b1b4bff3635 (DEFAULT_TABLE table=default.Table_500c159da8204c06b317eb5032e0113d [id=c22a16ca0ca649238ac510cb5542bd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.639515 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99090fee41d44946b6243b1b4bff3635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.639472 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d0cf3d979369419882a2ba2083933d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.641374 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e75b5fb51373451a94cb5ca59cdcefac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.641935 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet e75b5fb51373451a94cb5ca59cdcefac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.642668 885 tablet_service.cc:1468] Processing CreateTablet for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (DEFAULT_TABLE table=default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.643049 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.645516 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (DEFAULT_TABLE table=default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.645907 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.648694 893 tablet_service.cc:1468] Processing CreateTablet for tablet ce84f74470ad428abb2113efa170ca24 (DEFAULT_TABLE table=default.Table_aadd4b5489ea4e79b31a6d801a46cc87 [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.649117 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce84f74470ad428abb2113efa170ca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.650290 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ce84f74470ad428abb2113efa170ca24 (DEFAULT_TABLE table=default.Table_aadd4b5489ea4e79b31a6d801a46cc87 [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.650725 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce84f74470ad428abb2113efa170ca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.650929 1170 tablet_bootstrap.cc:492] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:52.651193 1170 ts_tablet_manager.cc:1397] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.282s user 0.021s sys 0.066s
I20250625 13:51:52.652181 1170 raft_consensus.cc:357] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.652519 1170 raft_consensus.cc:383] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.652659 1170 raft_consensus.cc:738] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.652859 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 969dc4a9b811496aa3d926a3ae3965ea (table default.Table_d64bbacfddeb4945b2396e41e59cbb23 [id=f00b1eb97d6245a8b22ad21216ed5230]) successfully deleted
W20250625 13:51:52.653148 581 catalog_manager.cc:4895] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 because the tablet was not found. No further retry: Not found: Tablet not found: 09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:52.653107 1170 consensus_queue.cc:260] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.654724 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d0cf3d979369419882a2ba2083933d48 (DEFAULT_TABLE table=default.Table_ac5c66505c43484ca1990febfc2a5bf9 [id=eb254e00e07047a09c1924a6cab20e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.655090 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cf3d979369419882a2ba2083933d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.655762 1170 ts_tablet_manager.cc:1428] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:52.660018 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 32221a413957450fa59ffcfbc91c51b9 (DEFAULT_TABLE table=default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.660472 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32221a413957450fa59ffcfbc91c51b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.661407 1170 tablet_bootstrap.cc:492] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:52.666587 1170 tablet_bootstrap.cc:654] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.669063 1184 raft_consensus.cc:491] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.669286 1184 raft_consensus.cc:513] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.670104 1184 leader_election.cc:290] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.672292 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b212f7c1ab48d2807a9eebeec31c07" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:52.672933 711 leader_election.cc:343] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.680388 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 921163dc5f454c9a97e72b005f9dba19 (DEFAULT_TABLE table=default.Table_04cf5db85d58479da934b19bc000f24d [id=96df25b43f664e9ebc958ff801d229b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.680768 754 tablet_service.cc:1468] Processing CreateTablet for tablet 921163dc5f454c9a97e72b005f9dba19 (DEFAULT_TABLE table=default.Table_04cf5db85d58479da934b19bc000f24d [id=96df25b43f664e9ebc958ff801d229b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.681209 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921163dc5f454c9a97e72b005f9dba19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.681515 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921163dc5f454c9a97e72b005f9dba19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.685678 1166 tablet_bootstrap.cc:492] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.685890 1166 ts_tablet_manager.cc:1397] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.320s user 0.028s sys 0.063s
I20250625 13:51:52.686640 1167 ts_tablet_manager.cc:1918] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.686791 1167 log.cc:1199] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/3dbeb984fadf4a0ebc50087c1a921f07
I20250625 13:51:52.686779 1166 raft_consensus.cc:357] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.687000 1166 raft_consensus.cc:383] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.687072 1166 raft_consensus.cc:738] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.687142 1167 ts_tablet_manager.cc:1939] T 3dbeb984fadf4a0ebc50087c1a921f07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.688498 778 raft_consensus.cc:1273] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer 1c008de91eb444f48da2c4f0b2d7a1ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:51:52.688331 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 3dbeb984fadf4a0ebc50087c1a921f07 (table default.Table_9875a118a9a74b7486cc83256370ddfd [id=72787a6701a64f08b90a8954cca4f049]) successfully deleted
W20250625 13:51:52.688923 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ca5dda8b27b34dafb20d8ed8afc58460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5dda8b27b34dafb20d8ed8afc58460 already in progress: creating tablet
W20250625 13:51:52.689141 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d because the tablet was not found. No further retry: Not found: Tablet not found: cf5a6951aba84f5fa8f6d76a2b7fcf2d
W20250625 13:51:52.689347 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d0cf3d979369419882a2ba2083933d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cf3d979369419882a2ba2083933d48 already in progress: creating tablet
I20250625 13:51:52.689265 1174 consensus_queue.cc:1035] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:52.690088 757 tablet_service.cc:1468] Processing CreateTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d (DEFAULT_TABLE table=default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e [id=2a70f0ade87f4f34bcbe23295f23234b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.690892 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.691053 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.690809 755 tablet_service.cc:1468] Processing CreateTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 (DEFAULT_TABLE table=default.Table_abfcbd249e4a4189aba6f16b5a66a0e7 [id=7d3d8e359ea945cc9495eb2454e1d85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.694015 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a8e15d57f14702ba0b5b52a94fc7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.696434 1165 ts_tablet_manager.cc:1918] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.700693 1165 log.cc:1199] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/e7906d7c2fb541dcb1ec8e8dcc729c25
I20250625 13:51:52.701179 1165 ts_tablet_manager.cc:1939] T e7906d7c2fb541dcb1ec8e8dcc729c25 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.702502 1165 tablet_replica.cc:331] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.702733 1165 raft_consensus.cc:2241] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.702981 1165 raft_consensus.cc:2270] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.703675 1165 ts_tablet_manager.cc:1905] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.699484 1166 consensus_queue.cc:260] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.702247 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet e7906d7c2fb541dcb1ec8e8dcc729c25 (table default.Table_c32eb049fff54229a1db62dc389476aa [id=bb31d8b38dd4479694c08744f757ef7d]) successfully deleted
I20250625 13:51:52.705494 1166 ts_tablet_manager.cc:1428] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
W20250625 13:51:52.705725 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e75b5fb51373451a94cb5ca59cdcefac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e75b5fb51373451a94cb5ca59cdcefac already in progress: creating tablet
W20250625 13:51:52.705969 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 because the tablet was not found. No further retry: Not found: Tablet not found: 09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:52.707485 1166 tablet_bootstrap.cc:492] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.711090 1021 tablet_service.cc:1468] Processing CreateTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d (DEFAULT_TABLE table=default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e [id=2a70f0ade87f4f34bcbe23295f23234b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.711596 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.713044 1166 tablet_bootstrap.cc:654] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.713442 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 (DEFAULT_TABLE table=default.Table_abfcbd249e4a4189aba6f16b5a66a0e7 [id=7d3d8e359ea945cc9495eb2454e1d85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.713876 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a8e15d57f14702ba0b5b52a94fc7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.718683 1165 ts_tablet_manager.cc:1918] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.725487 1249 tablet.cc:1722] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:51:52.725790 1165 log.cc:1199] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d455a2dec4304cf48ce29ceeb0700b0f
I20250625 13:51:52.726428 1165 ts_tablet_manager.cc:1939] T d455a2dec4304cf48ce29ceeb0700b0f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.727515 1165 tablet_replica.cc:331] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.727792 1165 raft_consensus.cc:2241] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.728019 1165 raft_consensus.cc:2270] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.728103 1244 mvcc.cc:204] Tried to move back new op lower bound from 7171520563930308608 to 7171520562089242624. Current Snapshot: MvccSnapshot[applied={T|T < 7171520563930308608}]
I20250625 13:51:52.728672 1165 ts_tablet_manager.cc:1905] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.730062 591 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6d92711284e64f71b31666800c250eab (table default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097]). Expected version 3 got 0
I20250625 13:51:52.730801 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_719d67c948c8495ba1f5c2001c020ce5" } modify_external_catalogs: true
I20250625 13:51:52.730974 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_719d67c948c8495ba1f5c2001c020ce5" } modify_external_catalogs: true
I20250625 13:51:52.731685 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 39dde16d7f144ef880b52abf9c979a53 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:52.732065 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99090fee41d44946b6243b1b4bff3635 consensus state has no leader
I20250625 13:51:52.732334 593 master_service.cc:948] ReplaceTablet: received request to replace tablet da4f9a46a5804da3aff1c9a87b2e4984 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:52.734136 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce84f74470ad428abb2113efa170ca24 consensus state has no leader
I20250625 13:51:52.734371 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 15750915ff7746ffa66509184be55dc4 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.734894 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a32960fbcf8043f69a0a57c8979a9b86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.735466 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a32960fbcf8043f69a0a57c8979a9b86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:52.736061 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 921163dc5f454c9a97e72b005f9dba19 consensus state has no leader
I20250625 13:51:52.736279 596 master_service.cc:948] ReplaceTablet: received request to replace tablet b4b212f7c1ab48d2807a9eebeec31c07 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.736810 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3599147b4b6b4aa3bcaa6de4475ae800" } modify_external_catalogs: true
I20250625 13:51:52.737003 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3599147b4b6b4aa3bcaa6de4475ae800" } modify_external_catalogs: true
I20250625 13:51:52.737135 1165 ts_tablet_manager.cc:1918] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.737267 1165 log.cc:1199] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a5afade7a5d1447a98934e4d614075d8
W20250625 13:51:52.737491 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da4f9a46a5804da3aff1c9a87b2e4984 consensus state has no leader
I20250625 13:51:52.737619 1165 ts_tablet_manager.cc:1939] T a5afade7a5d1447a98934e4d614075d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.737915 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d455a2dec4304cf48ce29ceeb0700b0f (table default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]) successfully deleted
I20250625 13:51:52.738541 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a5afade7a5d1447a98934e4d614075d8 (table default.Table_4afe38f204684075b394ad74549a04c2 [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
W20250625 13:51:52.739136 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09a8e15d57f14702ba0b5b52a94fc7c4 already in progress: creating tablet
I20250625 13:51:52.739444 1165 tablet_replica.cc:331] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.739609 1165 raft_consensus.cc:2241] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:52.740001 1165 pending_rounds.cc:70] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:52.740123 1165 pending_rounds.cc:77] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520562326781952 op_type: NO_OP noop_request { }
I20250625 13:51:52.740264 1165 raft_consensus.cc:2887] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:52.740360 1165 raft_consensus.cc:2270] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.741482 1165 ts_tablet_manager.cc:1905] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.745476 1248 tablet.cc:1722] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:51:52.749356 1165 ts_tablet_manager.cc:1918] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:52.749596 1165 log.cc:1199] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/c87280e6a3e84e6ba5b0a90e1e7ecec3
I20250625 13:51:52.750130 1165 ts_tablet_manager.cc:1939] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.750229 1170 tablet_bootstrap.cc:492] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:52.750397 1170 ts_tablet_manager.cc:1397] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
W20250625 13:51:52.750502 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39dde16d7f144ef880b52abf9c979a53 consensus state has no leader
I20250625 13:51:52.751688 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 60d6dcfd452448ecb242d863ee277170
I20250625 13:51:52.751283 1170 raft_consensus.cc:357] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.754029 1168 ts_tablet_manager.cc:1905] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.752044 1170 raft_consensus.cc:383] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.753116 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c282591e90c40948563fda06453e4d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 60d6dcfd452448ecb242d863ee277170) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.754673 1170 raft_consensus.cc:738] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.754858 1167 tablet_replica.cc:331] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.755040 1167 raft_consensus.cc:2241] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.755216 1167 raft_consensus.cc:2270] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.755478 1170 consensus_queue.cc:260] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.756062 1170 ts_tablet_manager.cc:1428] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:51:52.756296 1167 ts_tablet_manager.cc:1905] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.756359 1170 tablet_bootstrap.cc:492] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:52.757030 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b212f7c1ab48d2807a9eebeec31c07" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:52.758193 1170 tablet_bootstrap.cc:654] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.762059 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_34aa268b1b5c4293838192199719d473" } modify_external_catalogs: true
I20250625 13:51:52.762118 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1a06c7c260544cce91ab86dad2d62346"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.762315 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_34aa268b1b5c4293838192199719d473" } modify_external_catalogs: true
W20250625 13:51:52.762871 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a06c7c260544cce91ab86dad2d62346 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.767122 1167 ts_tablet_manager.cc:1918] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.767365 1167 log.cc:1199] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/3c282591e90c40948563fda06453e4d4
I20250625 13:51:52.767926 1167 ts_tablet_manager.cc:1939] T 3c282591e90c40948563fda06453e4d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.768986 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d8586363293040cc9ef927c2179bc1ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.769505 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8586363293040cc9ef927c2179bc1ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.770502 1165 tablet_replica.cc:331] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.770762 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 (table default.Table_d837899f74444b0a8e38179bb8dea2ee [id=bda1dd723374447b8a351345d7d3ae43]) successfully deleted
I20250625 13:51:52.771162 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 3c282591e90c40948563fda06453e4d4 (table default.Table_893b774d7d574f8e89a027212ab17b74 [id=18ba0d9199d14289bed69ea544d3dced]) successfully deleted
I20250625 13:51:52.772616 1166 tablet_bootstrap.cc:492] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.772820 1166 ts_tablet_manager.cc:1397] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.000s
I20250625 13:51:52.774010 1166 raft_consensus.cc:357] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.774240 1166 raft_consensus.cc:383] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.774456 1166 raft_consensus.cc:738] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.774749 1166 consensus_queue.cc:260] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.775190 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bf54376db991424b8de0e8f252fcb43f
I20250625 13:51:52.774358 1165 raft_consensus.cc:2241] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.776650 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet bf54376db991424b8de0e8f252fcb43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.776762 1165 raft_consensus.cc:2270] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.777741 755 tablet_service.cc:1515] Processing DeleteTablet for tablet bf54376db991424b8de0e8f252fcb43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.777081 1166 ts_tablet_manager.cc:1428] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250625 13:51:52.778831 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet da4f9a46a5804da3aff1c9a87b2e4984
I20250625 13:51:52.779271 593 catalog_manager.cc:6652] ReplaceTablet: tablet da4f9a46a5804da3aff1c9a87b2e4984 deleted and replaced by tablet bd5ba76c130a4711af5b21e9eb8562e7
I20250625 13:51:52.780076 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5ba76c130a4711af5b21e9eb8562e7) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.781031 755 tablet_service.cc:1515] Processing DeleteTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5ba76c130a4711af5b21e9eb8562e7) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.781287 1166 tablet_bootstrap.cc:492] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.782714 1165 ts_tablet_manager.cc:1905] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.785013 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a4578d0e3fe341039c02276aba0089fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.785564 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4578d0e3fe341039c02276aba0089fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.788410 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 99090fee41d44946b6243b1b4bff3635 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.788715 589 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097] alter complete (version 3)
I20250625 13:51:52.789917 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c282591e90c40948563fda06453e4d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 60d6dcfd452448ecb242d863ee277170) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.792948 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b4b212f7c1ab48d2807a9eebeec31c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.793757 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 96d491878b8343be93892aa594ca9da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.797765 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet dcb392fc4235435db39b0714b0270ee9
I20250625 13:51:52.798877 1178 raft_consensus.cc:491] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.799659 895 tablet_service.cc:1468] Processing CreateTablet for tablet d0cf3d979369419882a2ba2083933d48 (DEFAULT_TABLE table=default.Table_ac5c66505c43484ca1990febfc2a5bf9 [id=eb254e00e07047a09c1924a6cab20e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.801049 889 tablet_service.cc:1468] Processing CreateTablet for tablet 921163dc5f454c9a97e72b005f9dba19 (DEFAULT_TABLE table=default.Table_04cf5db85d58479da934b19bc000f24d [id=96df25b43f664e9ebc958ff801d229b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.801225 1171 tablet_bootstrap.cc:492] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.801386 891 tablet_service.cc:1515] Processing DeleteTablet for tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.801681 1178 raft_consensus.cc:513] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.802297 1178 leader_election.cc:290] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.802948 886 tablet_service.cc:1468] Processing CreateTablet for tablet 32221a413957450fa59ffcfbc91c51b9 (DEFAULT_TABLE table=default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.803275 894 tablet_service.cc:1468] Processing CreateTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d (DEFAULT_TABLE table=default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e [id=2a70f0ade87f4f34bcbe23295f23234b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.805359 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2273827d860d46af8da96df2ffd22a04" } modify_external_catalogs: true
I20250625 13:51:52.805614 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2273827d860d46af8da96df2ffd22a04" } modify_external_catalogs: true
I20250625 13:51:52.807109 896 tablet_service.cc:1515] Processing DeleteTablet for tablet e7906d7c2fb541dcb1ec8e8dcc729c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.807924 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c282591e90c40948563fda06453e4d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:52.810091 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cf3d979369419882a2ba2083933d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.810954 1171 ts_tablet_manager.cc:1397] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.304s user 0.021s sys 0.045s
I20250625 13:51:52.811254 915 raft_consensus.cc:1273] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Refusing update from remote peer 1c008de91eb444f48da2c4f0b2d7a1ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250625 13:51:52.812307 851 leader_election.cc:343] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.813067 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921163dc5f454c9a97e72b005f9dba19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.813449 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.813743 1166 tablet_bootstrap.cc:654] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.813905 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 96d491878b8343be93892aa594ca9da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.814646 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32221a413957450fa59ffcfbc91c51b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.815200 896 tablet_service.cc:1468] Processing CreateTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 (DEFAULT_TABLE table=default.Table_abfcbd249e4a4189aba6f16b5a66a0e7 [id=7d3d8e359ea945cc9495eb2454e1d85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.815644 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a8e15d57f14702ba0b5b52a94fc7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.816596 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.816763 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c282591e90c40948563fda06453e4d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 60d6dcfd452448ecb242d863ee277170) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.816916 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.818802 887 tablet_service.cc:1515] Processing DeleteTablet for tablet bf54376db991424b8de0e8f252fcb43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.819801 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b079a0f225cf4ac98c2efc03bb74b889"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.820391 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b079a0f225cf4ac98c2efc03bb74b889 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.821782 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f5662b7c3e9a430d87a09930ab6e8f5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.822304 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5662b7c3e9a430d87a09930ab6e8f5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.832408 1166 tablet_bootstrap.cc:492] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.832608 1166 ts_tablet_manager.cc:1397] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.001s
I20250625 13:51:52.833509 1166 raft_consensus.cc:357] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.833839 1166 raft_consensus.cc:383] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.835368 1166 raft_consensus.cc:738] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.838202 1166 consensus_queue.cc:260] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.835021 1171 raft_consensus.cc:357] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.838635 1171 raft_consensus.cc:383] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.838728 1171 raft_consensus.cc:738] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.838982 1171 consensus_queue.cc:260] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.839851 1166 ts_tablet_manager.cc:1428] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:51:52.840899 1166 tablet_bootstrap.cc:492] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.843072 1166 tablet_bootstrap.cc:654] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.845852 1165 ts_tablet_manager.cc:1918] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.846408 1165 log.cc:1199] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/b4b212f7c1ab48d2807a9eebeec31c07
I20250625 13:51:52.847220 1165 ts_tablet_manager.cc:1939] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.848510 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet b4b212f7c1ab48d2807a9eebeec31c07 (table default.Table_22c0e04fd41c41d981e66e94d5456772 [id=9b25f559cfc44e418b075045f83e64f2]) successfully deleted
W20250625 13:51:52.850214 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 15750915ff7746ffa66509184be55dc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15750915ff7746ffa66509184be55dc4 already in progress: creating tablet
W20250625 13:51:52.850909 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09a8e15d57f14702ba0b5b52a94fc7c4 already in progress: creating tablet
W20250625 13:51:52.851066 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ca5dda8b27b34dafb20d8ed8afc58460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5dda8b27b34dafb20d8ed8afc58460 already in progress: creating tablet
W20250625 13:51:52.851194 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d already in progress: creating tablet
W20250625 13:51:52.851334 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d0cf3d979369419882a2ba2083933d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cf3d979369419882a2ba2083933d48 already in progress: creating tablet
I20250625 13:51:52.852864 1165 tablet_replica.cc:331] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.853915 1165 raft_consensus.cc:2241] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.854790 1165 raft_consensus.cc:2270] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.856093 1166 tablet_bootstrap.cc:492] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.856244 1166 ts_tablet_manager.cc:1397] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250625 13:51:52.857064 1166 raft_consensus.cc:357] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.857228 1166 raft_consensus.cc:383] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.857321 1166 raft_consensus.cc:738] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.857545 1166 consensus_queue.cc:260] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.857730 1165 ts_tablet_manager.cc:1905] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.858919 1166 ts_tablet_manager.cc:1428] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:52.859603 1184 raft_consensus.cc:491] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.859773 1184 raft_consensus.cc:513] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.859874 1166 tablet_bootstrap.cc:492] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.861425 1184 leader_election.cc:290] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.862797 1166 tablet_bootstrap.cc:654] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:52.869666 711 leader_election.cc:343] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:52.869855 711 leader_election.cc:304] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:51:52.870107 711 raft_consensus.cc:2684] T b4b212f7c1ab48d2807a9eebeec31c07 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:51:52.870211 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76f074156559406c875e65796e6b5ee6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:52.870555 915 raft_consensus.cc:2466] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:52.875519 1166 tablet_bootstrap.cc:492] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:52.875739 1166 ts_tablet_manager.cc:1397] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.014s
W20250625 13:51:52.875924 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e75b5fb51373451a94cb5ca59cdcefac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e75b5fb51373451a94cb5ca59cdcefac already in progress: creating tablet
W20250625 13:51:52.876209 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bf54376db991424b8de0e8f252fcb43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf54376db991424b8de0e8f252fcb43f already in progress: creating tablet
W20250625 13:51:52.876451 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet da4f9a46a5804da3aff1c9a87b2e4984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da4f9a46a5804da3aff1c9a87b2e4984 already in progress: creating tablet
I20250625 13:51:52.876621 1166 raft_consensus.cc:357] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.876856 1166 raft_consensus.cc:383] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.876932 1166 raft_consensus.cc:738] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.877144 1171 ts_tablet_manager.cc:1428] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.066s user 0.001s sys 0.001s
I20250625 13:51:52.877207 1166 consensus_queue.cc:260] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.877555 1171 tablet_bootstrap.cc:492] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.878894 1166 ts_tablet_manager.cc:1428] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:51:52.879261 1171 tablet_bootstrap.cc:654] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.880138 1166 tablet_bootstrap.cc:492] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:52.882967 711 leader_election.cc:304] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:52.885286 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_186697102c744ac39ec9191fc0f992ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.885946 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_186697102c744ac39ec9191fc0f992ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.885957 1184 raft_consensus.cc:2802] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:52.886085 1184 raft_consensus.cc:491] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:52.886179 1184 raft_consensus.cc:3058] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.888276 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_aacf50f903c54d17a68e13722c931d22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.888448 1165 ts_tablet_manager.cc:1918] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:51:52.888974 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aacf50f903c54d17a68e13722c931d22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.888993 1165 log.cc:1199] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/96d491878b8343be93892aa594ca9da7
I20250625 13:51:52.889021 1184 raft_consensus.cc:513] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.889343 1220 raft_consensus.cc:491] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.889530 1220 raft_consensus.cc:513] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.890110 1220 leader_election.cc:290] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.891183 1184 leader_election.cc:290] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.891841 1165 ts_tablet_manager.cc:1939] T 96d491878b8343be93892aa594ca9da7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.891887 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dbeb984fadf4a0ebc50087c1a921f07" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:52.892117 916 raft_consensus.cc:2466] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:52.892621 711 leader_election.cc:304] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:52.893016 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76f074156559406c875e65796e6b5ee6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:52.893146 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 96d491878b8343be93892aa594ca9da7 (table default.Table_e42b0b5035ee4e30a0909702d04b052f [id=a364d50ed707453181496e70e26984e4]) successfully deleted
I20250625 13:51:52.893204 915 raft_consensus.cc:3058] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.896301 915 raft_consensus.cc:2466] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:52.896905 1184 raft_consensus.cc:2802] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:52.897060 1184 raft_consensus.cc:491] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:52.897217 1184 raft_consensus.cc:3058] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.900053 1184 raft_consensus.cc:513] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.900620 711 leader_election.cc:304] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:52.901041 1220 raft_consensus.cc:2802] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:52.901186 1220 raft_consensus.cc:695] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:52.901472 1220 consensus_queue.cc:237] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.902560 1171 tablet_bootstrap.cc:492] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:52.902773 1171 ts_tablet_manager.cc:1397] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250625 13:51:52.902809 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.905357 1165 tablet_replica.cc:331] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.905745 1168 ts_tablet_manager.cc:1918] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.905916 1168 log.cc:1199] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/b4b212f7c1ab48d2807a9eebeec31c07
I20250625 13:51:52.906353 1168 ts_tablet_manager.cc:1939] T b4b212f7c1ab48d2807a9eebeec31c07 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:52.906430 1165 raft_consensus.cc:2241] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.906519 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dbeb984fadf4a0ebc50087c1a921f07" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:52.906764 915 raft_consensus.cc:3058] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:52.907410 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet b4b212f7c1ab48d2807a9eebeec31c07 (table default.Table_22c0e04fd41c41d981e66e94d5456772 [id=9b25f559cfc44e418b075045f83e64f2]) successfully deleted
I20250625 13:51:52.907608 1168 tablet_replica.cc:331] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:52.907580 1165 raft_consensus.cc:2270] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:52.907686 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 15750915ff7746ffa66509184be55dc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15750915ff7746ffa66509184be55dc4 already in progress: creating tablet
I20250625 13:51:52.907800 1168 raft_consensus.cc:2241] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.907951 1168 raft_consensus.cc:2270] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.908809 889 tablet_service.cc:1515] Processing DeleteTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5ba76c130a4711af5b21e9eb8562e7) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.910495 915 raft_consensus.cc:2466] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:52.911517 1184 leader_election.cc:290] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:52.912163 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6d92711284e64f71b31666800c250eab
I20250625 13:51:52.903812 1171 raft_consensus.cc:357] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.913084 1171 raft_consensus.cc:383] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:52.913192 1171 raft_consensus.cc:738] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:52.913306 711 leader_election.cc:304] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:51:52.913491 711 raft_consensus.cc:2684] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:51:52.913414 1171 consensus_queue.cc:260] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.915746 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
I20250625 13:51:52.917186 1171 ts_tablet_manager.cc:1428] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250625 13:51:52.917524 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d92711284e64f71b31666800c250eab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.918018 754 tablet_service.cc:1468] Processing CreateTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 (DEFAULT_TABLE table=default.Table_7fdbbd3847ab4eb6b3f9872d5b0d9c4c [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.918501 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5786bb7716ef44689ef9862fc3ce17e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.920135 1171 tablet_bootstrap.cc:492] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:52.922716 1166 tablet_bootstrap.cc:654] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.923640 757 tablet_service.cc:1468] Processing CreateTablet for tablet cbaa018753d8455199f3869323fb5582 (DEFAULT_TABLE table=default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.923684 1197 consensus_queue.cc:1035] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.106s
I20250625 13:51:52.924021 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbaa018753d8455199f3869323fb5582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.921543 1171 tablet_bootstrap.cc:654] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:52.925451 1174 raft_consensus.cc:491] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:52.925875 1174 raft_consensus.cc:513] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:52.926193 1165 ts_tablet_manager.cc:1905] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.926635 1174 leader_election.cc:290] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:52.927030 758 tablet_service.cc:1468] Processing CreateTablet for tablet a21335a06ec04da29b3351554c75f03c (DEFAULT_TABLE table=default.Table_7a3fe0cc014e4cfd9e21d44e013bfc8b [id=3286dc94d1ba4645beb4a2a54bec8c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.929293 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76f074156559406c875e65796e6b5ee6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:52.929747 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76f074156559406c875e65796e6b5ee6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:52.930428 985 leader_election.cc:343] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b4b212f7c1ab48d2807a9eebeec31c07
I20250625 13:51:52.930800 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21335a06ec04da29b3351554c75f03c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:52.930970 849 leader_election.cc:343] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 3c282591e90c40948563fda06453e4d4
I20250625 13:51:52.931149 849 leader_election.cc:304] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:51:52.931887 711 leader_election.cc:343] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 3dbeb984fadf4a0ebc50087c1a921f07
W20250625 13:51:52.932232 711 leader_election.cc:343] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:52.932524 711 leader_election.cc:343] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 3dbeb984fadf4a0ebc50087c1a921f07
I20250625 13:51:52.932752 1178 raft_consensus.cc:2747] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:52.932804 711 leader_election.cc:343] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:52.933296 983 leader_election.cc:343] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: b4b212f7c1ab48d2807a9eebeec31c07
I20250625 13:51:52.933430 983 leader_election.cc:304] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:52.933843 1174 raft_consensus.cc:2747] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:52.933923 1167 tablet_replica.cc:331] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.934104 1167 raft_consensus.cc:2241] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:52.934315 1167 raft_consensus.cc:2270] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.935153 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 39dde16d7f144ef880b52abf9c979a53
I20250625 13:51:52.941397 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
I20250625 13:51:52.942370 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5b0408ce393341fb891ee6511dbf4890" } new_table_name: "default.Table_b13c5d5237f044c9bad038533aecc142" modify_external_catalogs: true
I20250625 13:51:52.942709 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 99090fee41d44946b6243b1b4bff3635
I20250625 13:51:52.942946 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0ae4786bc45842eea9376ff1f404d621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:52.943169 591 catalog_manager.cc:6652] ReplaceTablet: tablet 99090fee41d44946b6243b1b4bff3635 deleted and replaced by tablet d524089e6ab044ec961cfc9f587e8f86
W20250625 13:51:52.943531 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ae4786bc45842eea9376ff1f404d621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.943466 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4afe38f204684075b394ad74549a04c2" } new_table_name: "default.Table_a0708b1a41774b1c8ecdf97ea7e66531" modify_external_catalogs: true
I20250625 13:51:52.944856 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 99090fee41d44946b6243b1b4bff3635 with delete_type TABLET_DATA_DELETED (replaced by tablet d524089e6ab044ec961cfc9f587e8f86) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.945542 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ac6eba07f2c94c75a0be2404ab15f0d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:52.946071 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac6eba07f2c94c75a0be2404ab15f0d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:52.946419 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fb2e294ac01849e6bffc287498f9dcb3" } new_table_name: "default.Table_239d2674c30b4a6c9fe11163dc7f0b09" modify_external_catalogs: true
W20250625 13:51:52.948035 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bf54376db991424b8de0e8f252fcb43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf54376db991424b8de0e8f252fcb43f already in progress: creating tablet
W20250625 13:51:52.948272 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet da4f9a46a5804da3aff1c9a87b2e4984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da4f9a46a5804da3aff1c9a87b2e4984 already in progress: creating tablet
W20250625 13:51:52.948485 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3c282591e90c40948563fda06453e4d4 because the tablet was not found. No further retry: Not found: Tablet not found: 3c282591e90c40948563fda06453e4d4
I20250625 13:51:52.950076 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1481e562293146f799b1fbea03c28dc7" } new_table_name: "default.Table_01ec4a7ff073444b929c6c217632a150" modify_external_catalogs: true
I20250625 13:51:52.950379 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 (DEFAULT_TABLE table=default.Table_7fdbbd3847ab4eb6b3f9872d5b0d9c4c [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.950728 595 catalog_manager.cc:6652] ReplaceTablet: tablet 39dde16d7f144ef880b52abf9c979a53 deleted and replaced by tablet 8149aea0ff7646d3b94c9159f56e48ef
I20250625 13:51:52.950888 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5786bb7716ef44689ef9862fc3ce17e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.950956 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 39dde16d7f144ef880b52abf9c979a53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8149aea0ff7646d3b94c9159f56e48ef) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.953063 595 master_service.cc:948] ReplaceTablet: received request to replace tablet e75b5fb51373451a94cb5ca59cdcefac from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.953902 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1a7b72c2c464421eacfcf64008c3aea8
I20250625 13:51:52.955515 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:52.955745 1021 tablet_service.cc:1468] Processing CreateTablet for tablet cbaa018753d8455199f3869323fb5582 (DEFAULT_TABLE table=default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.956284 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbaa018753d8455199f3869323fb5582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.959630 1167 ts_tablet_manager.cc:1905] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:52.959690 1020 tablet_service.cc:1468] Processing CreateTablet for tablet a21335a06ec04da29b3351554c75f03c (DEFAULT_TABLE table=default.Table_7a3fe0cc014e4cfd9e21d44e013bfc8b [id=3286dc94d1ba4645beb4a2a54bec8c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.960225 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21335a06ec04da29b3351554c75f03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.960914 592 catalog_manager.cc:5582] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:51:52.961341 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
I20250625 13:51:52.963688 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 13620f6ca65342b4ad7f1ba4b437bde3 (DEFAULT_TABLE table=default.Table_6199817767f74031b8a500d21174d5a6 [id=bbf70badd2ea45ae88881afe31a3de40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.964186 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13620f6ca65342b4ad7f1ba4b437bde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.965605 1165 ts_tablet_manager.cc:1918] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.965749 1165 log.cc:1199] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/3dbeb984fadf4a0ebc50087c1a921f07
I20250625 13:51:52.966171 1165 ts_tablet_manager.cc:1939] T 3dbeb984fadf4a0ebc50087c1a921f07 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:52.967211 1165 tablet_replica.cc:331] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:52.967492 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 99090fee41d44946b6243b1b4bff3635 with delete_type TABLET_DATA_DELETED (replaced by tablet d524089e6ab044ec961cfc9f587e8f86) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.968688 747 tablet_service.cc:1468] Processing CreateTablet for tablet 2c1a98dc40684954a602567cf3476879 (DEFAULT_TABLE table=default.Table_18f2b0c4409b45848a64bb65f651c59e [id=97fc76973f074d29903fa25a80ca534a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.968737 750 tablet_service.cc:1468] Processing CreateTablet for tablet 7fe053e294fe4aceb24919e9e2bddf65 (DEFAULT_TABLE table=default.Table_d9dd2f12f8bb4f6cba435c892dac832e [id=98f8967275434f48a948def71ae1d693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.969231 753 tablet_service.cc:1468] Processing CreateTablet for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (DEFAULT_TABLE table=default.Table_ffbe3938117d49e0b9627a5c8f1d45f4 [id=a64872263d0d4aeaaac5a6df29cb73bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.969314 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1a98dc40684954a602567cf3476879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.969357 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d92711284e64f71b31666800c250eab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.969745 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ffbe3938117d49e0b9627a5c8f1d45f4" } new_table_name: "default.Table_c797297b30af49cf8e0b0f5d73a13a88" modify_external_catalogs: true
I20250625 13:51:52.970184 752 tablet_service.cc:1468] Processing CreateTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 (DEFAULT_TABLE table=default.Table_3599147b4b6b4aa3bcaa6de4475ae800 [id=cd3d67e4b72d4d919344482f7b10ed10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.970310 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 3dbeb984fadf4a0ebc50087c1a921f07 (table default.Table_9875a118a9a74b7486cc83256370ddfd [id=72787a6701a64f08b90a8954cca4f049]) successfully deleted
I20250625 13:51:52.971189 756 tablet_service.cc:1468] Processing CreateTablet for tablet 4b549bc8cead4af4a72f41f51655602c (DEFAULT_TABLE table=default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.971488 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_70ae19ea7742453484d9525b11f16752" } modify_external_catalogs: true
I20250625 13:51:52.971588 751 tablet_service.cc:1468] Processing CreateTablet for tablet 67441795e22b445a90c88d76aa519ea5 (DEFAULT_TABLE table=default.Table_fb2e294ac01849e6bffc287498f9dcb3 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.971689 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_70ae19ea7742453484d9525b11f16752" } modify_external_catalogs: true
I20250625 13:51:52.972008 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0f2fa5d5614ad394e6e12ec62814d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.972875 595 master_service.cc:948] ReplaceTablet: received request to replace tablet b11b3157dd764f229831c3b8417460c6 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:52.973172 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:52.973737 887 tablet_service.cc:1468] Processing CreateTablet for tablet 13620f6ca65342b4ad7f1ba4b437bde3 (DEFAULT_TABLE table=default.Table_6199817767f74031b8a500d21174d5a6 [id=bbf70badd2ea45ae88881afe31a3de40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.973973 892 tablet_service.cc:1468] Processing CreateTablet for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (DEFAULT_TABLE table=default.Table_ffbe3938117d49e0b9627a5c8f1d45f4 [id=a64872263d0d4aeaaac5a6df29cb73bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.974248 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13620f6ca65342b4ad7f1ba4b437bde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.974639 883 tablet_service.cc:1468] Processing CreateTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 (DEFAULT_TABLE table=default.Table_3599147b4b6b4aa3bcaa6de4475ae800 [id=cd3d67e4b72d4d919344482f7b10ed10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.975184 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7b72c2c464421eacfcf64008c3aea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.975427 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67441795e22b445a90c88d76aa519ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.976572 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2c1a98dc40684954a602567cf3476879 (DEFAULT_TABLE table=default.Table_18f2b0c4409b45848a64bb65f651c59e [id=97fc76973f074d29903fa25a80ca534a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.977066 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1a98dc40684954a602567cf3476879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.979322 891 tablet_service.cc:1468] Processing CreateTablet for tablet 7fe053e294fe4aceb24919e9e2bddf65 (DEFAULT_TABLE table=default.Table_d9dd2f12f8bb4f6cba435c892dac832e [id=98f8967275434f48a948def71ae1d693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.979615 890 tablet_service.cc:1468] Processing CreateTablet for tablet 67441795e22b445a90c88d76aa519ea5 (DEFAULT_TABLE table=default.Table_fb2e294ac01849e6bffc287498f9dcb3 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.979784 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe053e294fe4aceb24919e9e2bddf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.980125 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4b549bc8cead4af4a72f41f51655602c (DEFAULT_TABLE table=default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.980470 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b549bc8cead4af4a72f41f51655602c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.980640 1167 ts_tablet_manager.cc:1918] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:52.980790 1167 log.cc:1199] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/b4b212f7c1ab48d2807a9eebeec31c07
I20250625 13:51:52.981150 1167 ts_tablet_manager.cc:1939] T b4b212f7c1ab48d2807a9eebeec31c07 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:52.982281 1167 tablet_replica.cc:331] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:52.982453 1167 raft_consensus.cc:2241] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:52.982831 1167 raft_consensus.cc:2270] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:52.983207 755 tablet_service.cc:1468] Processing CreateTablet for tablet 13620f6ca65342b4ad7f1ba4b437bde3 (DEFAULT_TABLE table=default.Table_6199817767f74031b8a500d21174d5a6 [id=bbf70badd2ea45ae88881afe31a3de40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.983628 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (DEFAULT_TABLE table=default.Table_ffbe3938117d49e0b9627a5c8f1d45f4 [id=a64872263d0d4aeaaac5a6df29cb73bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.984033 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7b72c2c464421eacfcf64008c3aea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.984032 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0f2fa5d5614ad394e6e12ec62814d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.985111 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67441795e22b445a90c88d76aa519ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.987527 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e0f2fa5d5614ad394e6e12ec62814d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.987654 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 (DEFAULT_TABLE table=default.Table_3599147b4b6b4aa3bcaa6de4475ae800 [id=cd3d67e4b72d4d919344482f7b10ed10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.988013 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a7b72c2c464421eacfcf64008c3aea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.989323 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b549bc8cead4af4a72f41f51655602c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.991861 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7fe053e294fe4aceb24919e9e2bddf65 (DEFAULT_TABLE table=default.Table_d9dd2f12f8bb4f6cba435c892dac832e [id=98f8967275434f48a948def71ae1d693]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:52.992257 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe053e294fe4aceb24919e9e2bddf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:52.995504 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.995790 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 39dde16d7f144ef880b52abf9c979a53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8149aea0ff7646d3b94c9159f56e48ef) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:52.998987 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe053e294fe4aceb24919e9e2bddf65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.000960 894 tablet_service.cc:1468] Processing CreateTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 (DEFAULT_TABLE table=default.Table_7fdbbd3847ab4eb6b3f9872d5b0d9c4c [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.001403 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5786bb7716ef44689ef9862fc3ce17e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.001508 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13620f6ca65342b4ad7f1ba4b437bde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.002744 1165 raft_consensus.cc:2241] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.003049 1165 raft_consensus.cc:2270] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:53.003782 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67441795e22b445a90c88d76aa519ea5 consensus state has no leader
I20250625 13:51:53.007381 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 4b549bc8cead4af4a72f41f51655602c (DEFAULT_TABLE table=default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.007797 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b549bc8cead4af4a72f41f51655602c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.012279 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 2c1a98dc40684954a602567cf3476879 (DEFAULT_TABLE table=default.Table_18f2b0c4409b45848a64bb65f651c59e [id=97fc76973f074d29903fa25a80ca534a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.012701 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1a98dc40684954a602567cf3476879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.013756 1165 ts_tablet_manager.cc:1905] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.015656 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 67441795e22b445a90c88d76aa519ea5 (DEFAULT_TABLE table=default.Table_fb2e294ac01849e6bffc287498f9dcb3 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.016039 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67441795e22b445a90c88d76aa519ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.017397 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9e86330e77034242a9aa4fbd274c3151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.018179 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e86330e77034242a9aa4fbd274c3151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.019631 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a4578d0e3fe341039c02276aba0089fc" } modify_external_catalogs: true
I20250625 13:51:53.019860 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a4578d0e3fe341039c02276aba0089fc" } modify_external_catalogs: true
I20250625 13:51:53.031879 893 tablet_service.cc:1468] Processing CreateTablet for tablet a21335a06ec04da29b3351554c75f03c (DEFAULT_TABLE table=default.Table_7a3fe0cc014e4cfd9e21d44e013bfc8b [id=3286dc94d1ba4645beb4a2a54bec8c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.032464 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21335a06ec04da29b3351554c75f03c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:53.037408 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0723e91650174f5badd74f7756c5a54e consensus state has no leader
I20250625 13:51:53.037998 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f14230a4aed1468b82e23d0a1c081bad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.038748 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f14230a4aed1468b82e23d0a1c081bad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:53.056238 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd5ba76c130a4711af5b21e9eb8562e7 has no consensus state
I20250625 13:51:53.056568 590 master_service.cc:948] ReplaceTablet: received request to replace tablet ca5dda8b27b34dafb20d8ed8afc58460 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.057117 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_186697102c744ac39ec9191fc0f992ea" } modify_external_catalogs: true
I20250625 13:51:53.057308 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_186697102c744ac39ec9191fc0f992ea" } modify_external_catalogs: true
I20250625 13:51:53.057848 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7ccc261a6367449583a5c30495bfe2a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.058393 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ccc261a6367449583a5c30495bfe2a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.066336 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dbeb984fadf4a0ebc50087c1a921f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.066680 755 tablet_service.cc:1515] Processing DeleteTablet for tablet a418d2a6c8f24043a820967489c87e23 with delete_type TABLET_DATA_DELETED (replaced by tablet dcb392fc4235435db39b0714b0270ee9) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.067560 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c282591e90c40948563fda06453e4d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 60d6dcfd452448ecb242d863ee277170) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.074247 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet b4b212f7c1ab48d2807a9eebeec31c07 (table default.Table_22c0e04fd41c41d981e66e94d5456772 [id=9b25f559cfc44e418b075045f83e64f2]) successfully deleted
W20250625 13:51:53.074765 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 99090fee41d44946b6243b1b4bff3635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99090fee41d44946b6243b1b4bff3635 already in progress: creating tablet
I20250625 13:51:53.077543 896 tablet_service.cc:1468] Processing CreateTablet for tablet cbaa018753d8455199f3869323fb5582 (DEFAULT_TABLE table=default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.078197 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbaa018753d8455199f3869323fb5582. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:53.082803 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e0f2fa5d5614ad394e6e12ec62814d4 consensus state has no leader
I20250625 13:51:53.082944 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 39dde16d7f144ef880b52abf9c979a53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8149aea0ff7646d3b94c9159f56e48ef) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.083231 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d92711284e64f71b31666800c250eab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.089511 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 99090fee41d44946b6243b1b4bff3635 with delete_type TABLET_DATA_DELETED (replaced by tablet d524089e6ab044ec961cfc9f587e8f86) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.091351 1168 ts_tablet_manager.cc:1905] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:53.097887 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 921163dc5f454c9a97e72b005f9dba19 consensus state has no leader
I20250625 13:51:53.102025 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e930f596e0f6454c8af779d2e0f7ff76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.102674 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e930f596e0f6454c8af779d2e0f7ff76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.111433 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b11b3157dd764f229831c3b8417460c6
I20250625 13:51:53.111869 595 catalog_manager.cc:6652] ReplaceTablet: tablet b11b3157dd764f229831c3b8417460c6 deleted and replaced by tablet 85c72f9ae92a4537979fb479eb4467c4
I20250625 13:51:53.115535 755 tablet_service.cc:1515] Processing DeleteTablet for tablet b11b3157dd764f229831c3b8417460c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 85c72f9ae92a4537979fb479eb4467c4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.116958 1178 raft_consensus.cc:491] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.117206 1178 raft_consensus.cc:513] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.118045 1178 leader_election.cc:290] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.118440 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_85c33778738445ae9981ea2f1902e0ec" } modify_external_catalogs: true
I20250625 13:51:53.118643 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_85c33778738445ae9981ea2f1902e0ec" } modify_external_catalogs: true
I20250625 13:51:53.119603 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92a90b0dc85844a892ab247ae0d09a12" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:53.119973 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92a90b0dc85844a892ab247ae0d09a12" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:53.120258 777 raft_consensus.cc:2466] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:53.120930 851 leader_election.cc:304] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:53.121387 1178 raft_consensus.cc:2802] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:53.121501 1178 raft_consensus.cc:491] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:53.121618 1178 raft_consensus.cc:3058] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.125267 1178 raft_consensus.cc:513] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.126336 593 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet b52c814dc55548e6b309f222e4f049c7
I20250625 13:51:53.126729 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92a90b0dc85844a892ab247ae0d09a12" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:53.126995 777 raft_consensus.cc:3058] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.130775 777 raft_consensus.cc:2466] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:53.131639 851 leader_election.cc:304] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:53.133636 1050 raft_consensus.cc:2466] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:53.142920 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1be3d9db61ab4b56b98b70af09df98b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.143782 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1be3d9db61ab4b56b98b70af09df98b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.145402 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4212481baebd45429d103db60e844de5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.146131 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4212481baebd45429d103db60e844de5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.146126 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92a90b0dc85844a892ab247ae0d09a12" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:53.146719 1178 leader_election.cc:290] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.146972 1178 raft_consensus.cc:2802] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:53.147118 1178 raft_consensus.cc:695] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:53.147399 1178 consensus_queue.cc:237] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.147706 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_eb08aa18af9641049c9d7345375fa250"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.148352 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb08aa18af9641049c9d7345375fa250 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.148901 883 tablet_service.cc:1515] Processing DeleteTablet for tablet b11b3157dd764f229831c3b8417460c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 85c72f9ae92a4537979fb479eb4467c4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.153931 1050 raft_consensus.cc:3058] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.168167 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5964baca5b4a4701bf5a1b86d494b4e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.169139 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5964baca5b4a4701bf5a1b86d494b4e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.169399 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet b11b3157dd764f229831c3b8417460c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 85c72f9ae92a4537979fb479eb4467c4) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.177027 1171 tablet_bootstrap.cc:492] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.177223 1171 ts_tablet_manager.cc:1397] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.257s user 0.030s sys 0.052s
I20250625 13:51:53.178085 1171 raft_consensus.cc:357] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.178303 1171 raft_consensus.cc:383] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.178365 1171 raft_consensus.cc:738] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.178571 1171 consensus_queue.cc:260] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.179216 1050 raft_consensus.cc:2466] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:53.179603 1165 ts_tablet_manager.cc:1918] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:51:53.179807 1166 tablet_bootstrap.cc:492] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.179831 1165 log.cc:1199] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/6d92711284e64f71b31666800c250eab
I20250625 13:51:53.180418 1165 ts_tablet_manager.cc:1939] T 6d92711284e64f71b31666800c250eab P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:53.179988 1166 ts_tablet_manager.cc:1397] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.300s user 0.031s sys 0.053s
I20250625 13:51:53.182086 1165 tablet_replica.cc:331] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:53.182336 1165 raft_consensus.cc:2241] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.182593 1165 raft_consensus.cc:2270] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.183259 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 6d92711284e64f71b31666800c250eab (table default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097]) successfully deleted
W20250625 13:51:53.183634 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 99090fee41d44946b6243b1b4bff3635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99090fee41d44946b6243b1b4bff3635 already in progress: creating tablet
I20250625 13:51:53.183789 1165 ts_tablet_manager.cc:1905] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:53.183871 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 39dde16d7f144ef880b52abf9c979a53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39dde16d7f144ef880b52abf9c979a53 already in progress: creating tablet
W20250625 13:51:53.184059 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1a7b72c2c464421eacfcf64008c3aea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a7b72c2c464421eacfcf64008c3aea8 already in progress: creating tablet
W20250625 13:51:53.184263 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3dbeb984fadf4a0ebc50087c1a921f07 because the tablet was not found. No further retry: Not found: Tablet not found: 3dbeb984fadf4a0ebc50087c1a921f07
I20250625 13:51:53.182152 1166 raft_consensus.cc:357] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.184548 1166 raft_consensus.cc:383] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.184687 1166 raft_consensus.cc:738] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.184948 1166 consensus_queue.cc:260] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.189996 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_888aed1788b245e5b873ac73a0dcfba5" } modify_external_catalogs: true
I20250625 13:51:53.190194 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_888aed1788b245e5b873ac73a0dcfba5" } modify_external_catalogs: true
I20250625 13:51:53.190265 1166 ts_tablet_manager.cc:1428] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250625 13:51:53.190960 1165 ts_tablet_manager.cc:1918] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.193009 1166 tablet_bootstrap.cc:492] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.197224 1165 log.cc:1199] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a418d2a6c8f24043a820967489c87e23
I20250625 13:51:53.197942 1165 ts_tablet_manager.cc:1939] T a418d2a6c8f24043a820967489c87e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:53.199294 1166 tablet_bootstrap.cc:654] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.199774 1165 tablet_replica.cc:331] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:53.200059 1165 raft_consensus.cc:2241] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.201236 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a418d2a6c8f24043a820967489c87e23 (table default.Table_34aa268b1b5c4293838192199719d473 [id=1ab84062698540c38a1047d6ce6476da]) successfully deleted
I20250625 13:51:53.201479 1165 raft_consensus.cc:2270] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.202419 1165 ts_tablet_manager.cc:1905] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.205493 1170 tablet_bootstrap.cc:492] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.205683 1170 ts_tablet_manager.cc:1397] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.449s user 0.110s sys 0.154s
I20250625 13:51:53.207093 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 32221a413957450fa59ffcfbc91c51b9 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.207031 1170 raft_consensus.cc:357] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.207278 1170 raft_consensus.cc:383] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.207386 1170 raft_consensus.cc:738] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.207669 1170 consensus_queue.cc:260] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.208245 1170 ts_tablet_manager.cc:1428] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:53.208580 1170 tablet_bootstrap.cc:492] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.209918 755 tablet_service.cc:1515] Processing DeleteTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.211534 1170 tablet_bootstrap.cc:654] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:53.213096 1243 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:42238: Illegal state: RaftConsensus is not running: State = Stopped
I20250625 13:51:53.213447 1165 ts_tablet_manager.cc:1918] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.213663 1165 log.cc:1199] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/3c282591e90c40948563fda06453e4d4
I20250625 13:51:53.214115 1165 ts_tablet_manager.cc:1939] T 3c282591e90c40948563fda06453e4d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:53.215154 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 3c282591e90c40948563fda06453e4d4 (table default.Table_893b774d7d574f8e89a027212ab17b74 [id=18ba0d9199d14289bed69ea544d3dced]) successfully deleted
I20250625 13:51:53.216428 1165 tablet_replica.cc:331] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:53.216615 1165 raft_consensus.cc:2241] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:53.216614 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet b11b3157dd764f229831c3b8417460c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b11b3157dd764f229831c3b8417460c6 already in progress: creating tablet
I20250625 13:51:53.216805 1165 raft_consensus.cc:2270] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.219903 1167 ts_tablet_manager.cc:1905] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:53.224470 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 92884.9us
I20250625 13:51:53.227278 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 872724cd95bf473cb40e5f8ed8b5bd55 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.228940 596 master_service.cc:948] ReplaceTablet: received request to replace tablet ca5dda8b27b34dafb20d8ed8afc58460 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.229429 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a32960fbcf8043f69a0a57c8979a9b86" } new_table_name: "default.Table_b5472b871d6743cb80ba576b1cf02ee6" modify_external_catalogs: true
I20250625 13:51:53.229607 1167 ts_tablet_manager.cc:1918] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:51:53.229761 1167 log.cc:1199] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/6d92711284e64f71b31666800c250eab
I20250625 13:51:53.230001 1165 ts_tablet_manager.cc:1905] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.230203 1167 ts_tablet_manager.cc:1939] T 6d92711284e64f71b31666800c250eab P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:53.231206 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 6d92711284e64f71b31666800c250eab (table default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097]) successfully deleted
I20250625 13:51:53.232252 1170 tablet_bootstrap.cc:492] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.232523 1170 ts_tablet_manager.cc:1397] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
W20250625 13:51:53.234174 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1a7b72c2c464421eacfcf64008c3aea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a7b72c2c464421eacfcf64008c3aea8 already in progress: creating tablet
I20250625 13:51:53.235172 1170 raft_consensus.cc:357] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:53.236107 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 39dde16d7f144ef880b52abf9c979a53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39dde16d7f144ef880b52abf9c979a53 already in progress: creating tablet
W20250625 13:51:53.236929 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b11b3157dd764f229831c3b8417460c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b11b3157dd764f229831c3b8417460c6 already in progress: creating tablet
I20250625 13:51:53.237242 1170 raft_consensus.cc:383] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.238243 1170 raft_consensus.cc:738] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.238603 1170 consensus_queue.cc:260] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.238615 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_015b4a77da244eaf9d34a281215e77c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.239245 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_015b4a77da244eaf9d34a281215e77c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.239348 1170 ts_tablet_manager.cc:1428] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 13:51:53.239671 1170 tablet_bootstrap.cc:492] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.241518 1170 tablet_bootstrap.cc:654] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.242539 1168 ts_tablet_manager.cc:1918] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.245072 1168 log.cc:1199] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a418d2a6c8f24043a820967489c87e23
I20250625 13:51:53.245589 1168 ts_tablet_manager.cc:1939] T a418d2a6c8f24043a820967489c87e23 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:53.245865 1171 ts_tablet_manager.cc:1428] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.069s user 0.001s sys 0.007s
I20250625 13:51:53.247103 1171 tablet_bootstrap.cc:492] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.248736 1171 tablet_bootstrap.cc:654] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.249989 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c797297b30af49cf8e0b0f5d73a13a88" } new_table_name: "default.Table_675c4ec332bd4560b7e40c2bc7ce5541" modify_external_catalogs: true
W20250625 13:51:53.251255 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09a8e15d57f14702ba0b5b52a94fc7c4 already in progress: creating tablet
I20250625 13:51:53.253367 1266 tablet.cc:1722] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:51:53.259373 1166 tablet_bootstrap.cc:492] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
W20250625 13:51:53.259675 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ca5dda8b27b34dafb20d8ed8afc58460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5dda8b27b34dafb20d8ed8afc58460 already in progress: creating tablet
I20250625 13:51:53.259712 1166 ts_tablet_manager.cc:1397] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.001s
W20250625 13:51:53.260448 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d already in progress: creating tablet
I20250625 13:51:53.261533 1166 raft_consensus.cc:357] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.261790 1166 raft_consensus.cc:383] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.261873 1166 raft_consensus.cc:738] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.262158 1166 consensus_queue.cc:260] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.264844 1166 ts_tablet_manager.cc:1428] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:51:53.265164 1166 tablet_bootstrap.cc:492] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.269481 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
I20250625 13:51:53.269685 1168 tablet_replica.cc:331] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:53.269913 1168 raft_consensus.cc:2241] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.270115 1168 raft_consensus.cc:2270] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.270717 1168 ts_tablet_manager.cc:1905] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:53.274799 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d0cf3d979369419882a2ba2083933d48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0cf3d979369419882a2ba2083933d48 already in progress: creating tablet
I20250625 13:51:53.275126 1165 ts_tablet_manager.cc:1918] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:51:53.275179 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e75b5fb51373451a94cb5ca59cdcefac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e75b5fb51373451a94cb5ca59cdcefac already in progress: creating tablet
I20250625 13:51:53.275264 1165 log.cc:1199] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/b44c85b12fe64fc99fcf0a79a5909c6c
I20250625 13:51:53.275655 1166 tablet_bootstrap.cc:654] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.276026 893 tablet_service.cc:1468] Processing CreateTablet for tablet 02f2a2b595bc43309a29b3f1bc5d3766 (DEFAULT_TABLE table=default.Table_aacf50f903c54d17a68e13722c931d22 [id=8c054766f5424144a4edb39b192451cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.276480 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f2a2b595bc43309a29b3f1bc5d3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.277257 884 tablet_service.cc:1468] Processing CreateTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee (DEFAULT_TABLE table=default.Table_d8586363293040cc9ef927c2179bc1ae [id=09a0b05c11214fd0952325f08a002b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.277459 1171 tablet_bootstrap.cc:492] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.277642 1171 ts_tablet_manager.cc:1397] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20250625 13:51:53.277698 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14a12a5453e647b9946ca19e0ffb14ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.278499 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 02f2a2b595bc43309a29b3f1bc5d3766 (DEFAULT_TABLE table=default.Table_aacf50f903c54d17a68e13722c931d22 [id=8c054766f5424144a4edb39b192451cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.278499 1171 raft_consensus.cc:357] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.278728 1171 raft_consensus.cc:383] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.278821 1171 raft_consensus.cc:738] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.278965 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f2a2b595bc43309a29b3f1bc5d3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.281013 754 tablet_service.cc:1468] Processing CreateTablet for tablet 36da1da539e24f55978d516873cb0b27 (DEFAULT_TABLE table=default.Table_b079a0f225cf4ac98c2efc03bb74b889 [id=83a712e3adf74a95a5c2ea3fc5c0e14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.281423 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36da1da539e24f55978d516873cb0b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.281471 752 tablet_service.cc:1468] Processing CreateTablet for tablet b7641869a0bb463698244ff445d87a1f (DEFAULT_TABLE table=default.Table_6e5d1377cfc84651afd53a78735468d8 [id=bd476a97393841899b4cdb426c39e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.281646 755 tablet_service.cc:1468] Processing CreateTablet for tablet 02f2a2b595bc43309a29b3f1bc5d3766 (DEFAULT_TABLE table=default.Table_aacf50f903c54d17a68e13722c931d22 [id=8c054766f5424144a4edb39b192451cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.281836 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7641869a0bb463698244ff445d87a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.282212 747 tablet_service.cc:1468] Processing CreateTablet for tablet 07e281786e6e4a5d93b94b85b486655d (DEFAULT_TABLE table=default.Table_f5662b7c3e9a430d87a09930ab6e8f5c [id=7b67603755e7428496524ee716b435e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.282575 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e281786e6e4a5d93b94b85b486655d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.283597 757 tablet_service.cc:1468] Processing CreateTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 (DEFAULT_TABLE table=default.Table_a32960fbcf8043f69a0a57c8979a9b86 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.283630 883 tablet_service.cc:1468] Processing CreateTablet for tablet 600d711975a74f66b9144758ad0dcfb0 (DEFAULT_TABLE table=default.Table_186697102c744ac39ec9191fc0f992ea [id=548110492fba410e9612ddd9792eed22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.283950 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600d711975a74f66b9144758ad0dcfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.283977 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87327f74beb845768bbfd5ae6d2bcf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.277256 894 tablet_service.cc:1468] Processing CreateTablet for tablet d4f19ddc51324cc18242c6081f3f7de5 (DEFAULT_TABLE table=default.Table_969aea61c59149e39d51efed86647070 [id=b8285b64bbee49f9be8741b252d03363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.285526 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f19ddc51324cc18242c6081f3f7de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.287638 756 tablet_service.cc:1468] Processing CreateTablet for tablet 600d711975a74f66b9144758ad0dcfb0 (DEFAULT_TABLE table=default.Table_186697102c744ac39ec9191fc0f992ea [id=548110492fba410e9612ddd9792eed22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.287644 895 tablet_service.cc:1468] Processing CreateTablet for tablet 36da1da539e24f55978d516873cb0b27 (DEFAULT_TABLE table=default.Table_b079a0f225cf4ac98c2efc03bb74b889 [id=83a712e3adf74a95a5c2ea3fc5c0e14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.287999 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36da1da539e24f55978d516873cb0b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.288025 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600d711975a74f66b9144758ad0dcfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.290074 891 tablet_service.cc:1468] Processing CreateTablet for tablet b7641869a0bb463698244ff445d87a1f (DEFAULT_TABLE table=default.Table_6e5d1377cfc84651afd53a78735468d8 [id=bd476a97393841899b4cdb426c39e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.290427 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7641869a0bb463698244ff445d87a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.291673 753 tablet_service.cc:1468] Processing CreateTablet for tablet 389d81329b8d430a9098b0630612fdcb (DEFAULT_TABLE table=default.Table_1a06c7c260544cce91ab86dad2d62346 [id=aff513ce6735496784852aef3728d643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.292057 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 389d81329b8d430a9098b0630612fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.294052 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 07e281786e6e4a5d93b94b85b486655d (DEFAULT_TABLE table=default.Table_f5662b7c3e9a430d87a09930ab6e8f5c [id=7b67603755e7428496524ee716b435e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.294394 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e281786e6e4a5d93b94b85b486655d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.294358 1171 consensus_queue.cc:260] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.294957 1171 ts_tablet_manager.cc:1428] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20250625 13:51:53.295573 751 tablet_service.cc:1468] Processing CreateTablet for tablet d524089e6ab044ec961cfc9f587e8f86 (DEFAULT_TABLE table=default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.295591 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 600d711975a74f66b9144758ad0dcfb0 (DEFAULT_TABLE table=default.Table_186697102c744ac39ec9191fc0f992ea [id=548110492fba410e9612ddd9792eed22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.295898 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 600d711975a74f66b9144758ad0dcfb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.295953 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d524089e6ab044ec961cfc9f587e8f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.299598 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 389d81329b8d430a9098b0630612fdcb (DEFAULT_TABLE table=default.Table_1a06c7c260544cce91ab86dad2d62346 [id=aff513ce6735496784852aef3728d643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.299726 1165 ts_tablet_manager.cc:1939] T b44c85b12fe64fc99fcf0a79a5909c6c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:53.299649 758 tablet_service.cc:1468] Processing CreateTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee (DEFAULT_TABLE table=default.Table_d8586363293040cc9ef927c2179bc1ae [id=09a0b05c11214fd0952325f08a002b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.299995 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 389d81329b8d430a9098b0630612fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.300127 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14a12a5453e647b9946ca19e0ffb14ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.303519 1171 tablet_bootstrap.cc:492] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.303644 750 tablet_service.cc:1468] Processing CreateTablet for tablet d4f19ddc51324cc18242c6081f3f7de5 (DEFAULT_TABLE table=default.Table_969aea61c59149e39d51efed86647070 [id=b8285b64bbee49f9be8741b252d03363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.303718 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 (DEFAULT_TABLE table=default.Table_a32960fbcf8043f69a0a57c8979a9b86 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.304090 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87327f74beb845768bbfd5ae6d2bcf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.304104 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f19ddc51324cc18242c6081f3f7de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.305819 1171 tablet_bootstrap.cc:654] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.307605 1021 tablet_service.cc:1468] Processing CreateTablet for tablet d524089e6ab044ec961cfc9f587e8f86 (DEFAULT_TABLE table=default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.308050 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f2a2b595bc43309a29b3f1bc5d3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.308089 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d524089e6ab044ec961cfc9f587e8f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.311622 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b7641869a0bb463698244ff445d87a1f (DEFAULT_TABLE table=default.Table_6e5d1377cfc84651afd53a78735468d8 [id=bd476a97393841899b4cdb426c39e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.312026 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7641869a0bb463698244ff445d87a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.312238 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet b44c85b12fe64fc99fcf0a79a5909c6c (table default.Table_3f5b9820585e4ed89aa7d0abca37a52c [id=d7e8d11b88e240b89d40754b67570894]) successfully deleted
I20250625 13:51:53.315785 1027 tablet_service.cc:1468] Processing CreateTablet for tablet d4f19ddc51324cc18242c6081f3f7de5 (DEFAULT_TABLE table=default.Table_969aea61c59149e39d51efed86647070 [id=b8285b64bbee49f9be8741b252d03363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.316293 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f19ddc51324cc18242c6081f3f7de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.319952 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee (DEFAULT_TABLE table=default.Table_d8586363293040cc9ef927c2179bc1ae [id=09a0b05c11214fd0952325f08a002b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.320068 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_efbacb69562d4fe484a0cafa47e8985e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:53.320452 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14a12a5453e647b9946ca19e0ffb14ee. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:53.320660 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efbacb69562d4fe484a0cafa47e8985e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.323179 752 tablet_service.cc:1515] Processing DeleteTablet for tablet a17f862a8a584047a8f0220f2eced869 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.323487 1165 tablet_replica.cc:331] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:53.323727 1165 raft_consensus.cc:2241] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.323921 1165 raft_consensus.cc:2270] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.324743 1165 ts_tablet_manager.cc:1905] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.325068 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet a418d2a6c8f24043a820967489c87e23 with delete_type TABLET_DATA_DELETED (replaced by tablet dcb392fc4235435db39b0714b0270ee9) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.325502 1167 tablet_replica.cc:331] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:53.325690 1167 raft_consensus.cc:2241] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.325862 1167 raft_consensus.cc:2270] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.326802 889 tablet_service.cc:1468] Processing CreateTablet for tablet 389d81329b8d430a9098b0630612fdcb (DEFAULT_TABLE table=default.Table_1a06c7c260544cce91ab86dad2d62346 [id=aff513ce6735496784852aef3728d643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.327225 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 389d81329b8d430a9098b0630612fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.330914 887 tablet_service.cc:1468] Processing CreateTablet for tablet d524089e6ab044ec961cfc9f587e8f86 (DEFAULT_TABLE table=default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.331347 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d524089e6ab044ec961cfc9f587e8f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.334280 886 tablet_service.cc:1468] Processing CreateTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 (DEFAULT_TABLE table=default.Table_a32960fbcf8043f69a0a57c8979a9b86 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.334535 1165 ts_tablet_manager.cc:1918] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.334712 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87327f74beb845768bbfd5ae6d2bcf88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.334712 1165 log.cc:1199] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a17f862a8a584047a8f0220f2eced869
I20250625 13:51:53.335091 1165 ts_tablet_manager.cc:1939] T a17f862a8a584047a8f0220f2eced869 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:53.337492 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a17f862a8a584047a8f0220f2eced869 (table default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]) successfully deleted
I20250625 13:51:53.337739 1178 raft_consensus.cc:491] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.337873 1178 raft_consensus.cc:513] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.338548 1178 leader_election.cc:290] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.338697 1178 raft_consensus.cc:491] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.339730 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96d491878b8343be93892aa594ca9da7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:53.299263 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 36da1da539e24f55978d516873cb0b27 (DEFAULT_TABLE table=default.Table_b079a0f225cf4ac98c2efc03bb74b889 [id=83a712e3adf74a95a5c2ea3fc5c0e14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.340711 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36da1da539e24f55978d516873cb0b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.342325 1029 tablet_service.cc:1468] Processing CreateTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 (DEFAULT_TABLE table=default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.342701 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5ba76c130a4711af5b21e9eb8562e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.342793 758 tablet_service.cc:1468] Processing CreateTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef (DEFAULT_TABLE table=default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.343178 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8149aea0ff7646d3b94c9159f56e48ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.344827 1167 ts_tablet_manager.cc:1905] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:53.353036 851 leader_election.cc:343] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 96d491878b8343be93892aa594ca9da7
W20250625 13:51:53.353688 849 leader_election.cc:343] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:53.354097 884 tablet_service.cc:1468] Processing CreateTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 (DEFAULT_TABLE table=default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.355566 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5ba76c130a4711af5b21e9eb8562e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.356532 1178 raft_consensus.cc:513] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.356772 894 tablet_service.cc:1468] Processing CreateTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef (DEFAULT_TABLE table=default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.357122 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8149aea0ff7646d3b94c9159f56e48ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.357136 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef (DEFAULT_TABLE table=default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.357614 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8149aea0ff7646d3b94c9159f56e48ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.358161 1178 leader_election.cc:290] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:51:53.359146 851 leader_election.cc:343] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a17f862a8a584047a8f0220f2eced869
I20250625 13:51:53.359388 849 leader_election.cc:304] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:51:53.362026 849 leader_election.cc:343] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: a17f862a8a584047a8f0220f2eced869
I20250625 13:51:53.362162 849 leader_election.cc:304] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:53.362668 1178 raft_consensus.cc:2747] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:53.362852 1178 raft_consensus.cc:2747] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:53.363101 896 tablet_service.cc:1468] Processing CreateTablet for tablet 07e281786e6e4a5d93b94b85b486655d (DEFAULT_TABLE table=default.Table_f5662b7c3e9a430d87a09930ab6e8f5c [id=7b67603755e7428496524ee716b435e9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:53.366384 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ce84f74470ad428abb2113efa170ca24 (table default.Table_2c63842cb66d43c19e91b079c23ad1df [id=1ccdf11a8fe64810a774e47fb27d0916], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ce84f74470ad428abb2113efa170ca24 consensus state has no leader
I20250625 13:51:53.368806 750 tablet_service.cc:1468] Processing CreateTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 (DEFAULT_TABLE table=default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.369228 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5ba76c130a4711af5b21e9eb8562e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:53.379858 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87327f74beb845768bbfd5ae6d2bcf88 consensus state has no leader
I20250625 13:51:53.380926 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e281786e6e4a5d93b94b85b486655d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.385655 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a0708b1a41774b1c8ecdf97ea7e66531" } new_table_name: "default.Table_bcac7658498546e39e2b1f2d30910433" modify_external_catalogs: true
I20250625 13:51:53.388605 758 tablet_service.cc:1515] Processing DeleteTablet for tablet a17f862a8a584047a8f0220f2eced869 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.394927 593 master_service.cc:948] ReplaceTablet: received request to replace tablet cbaa018753d8455199f3869323fb5582 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:53.405943 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e0f2fa5d5614ad394e6e12ec62814d4 consensus state has no leader
I20250625 13:51:53.410677 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_18f2b0c4409b45848a64bb65f651c59e" } new_table_name: "default.Table_1231c5f6367b4046beaa57696ea42ff4" modify_external_catalogs: true
I20250625 13:51:53.416178 1167 ts_tablet_manager.cc:1918] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.416491 1167 log.cc:1199] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a418d2a6c8f24043a820967489c87e23
I20250625 13:51:53.416986 1167 ts_tablet_manager.cc:1939] T a418d2a6c8f24043a820967489c87e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:53.418215 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a418d2a6c8f24043a820967489c87e23 (table default.Table_34aa268b1b5c4293838192199719d473 [id=1ab84062698540c38a1047d6ce6476da]) successfully deleted
I20250625 13:51:53.419601 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 32221a413957450fa59ffcfbc91c51b9
I20250625 13:51:53.420082 589 catalog_manager.cc:6652] ReplaceTablet: tablet 32221a413957450fa59ffcfbc91c51b9 deleted and replaced by tablet bb1c7a85da844d34be805843bc60d350
I20250625 13:51:53.422793 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 32221a413957450fa59ffcfbc91c51b9 with delete_type TABLET_DATA_DELETED (replaced by tablet bb1c7a85da844d34be805843bc60d350) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:53.423483 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a17f862a8a584047a8f0220f2eced869 because the tablet was not found. No further retry: Not found: Tablet not found: a17f862a8a584047a8f0220f2eced869
I20250625 13:51:53.423786 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 32221a413957450fa59ffcfbc91c51b9 with delete_type TABLET_DATA_DELETED (replaced by tablet bb1c7a85da844d34be805843bc60d350) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.426705 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_969aea61c59149e39d51efed86647070" } new_table_name: "default.Table_f2cf2aab62814c7389fa4b446b696dd1" modify_external_catalogs: true
W20250625 13:51:53.430153 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 32221a413957450fa59ffcfbc91c51b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32221a413957450fa59ffcfbc91c51b9 already in progress: creating tablet
W20250625 13:51:53.435842 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 32221a413957450fa59ffcfbc91c51b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32221a413957450fa59ffcfbc91c51b9 already in progress: creating tablet
I20250625 13:51:53.436909 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 40a6375e98a04f95aa2bbde6b9b0228f
I20250625 13:51:53.441404 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:53.442418 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 40a6375e98a04f95aa2bbde6b9b0228f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40a6375e98a04f95aa2bbde6b9b0228f already in progress: creating tablet
I20250625 13:51:53.443538 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.443656 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f14230a4aed1468b82e23d0a1c081bad" } modify_external_catalogs: true
I20250625 13:51:53.443930 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.443915 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f14230a4aed1468b82e23d0a1c081bad" } modify_external_catalogs: true
W20250625 13:51:53.444613 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 40a6375e98a04f95aa2bbde6b9b0228f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40a6375e98a04f95aa2bbde6b9b0228f already in progress: creating tablet
I20250625 13:51:53.444972 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 32221a413957450fa59ffcfbc91c51b9 with delete_type TABLET_DATA_DELETED (replaced by tablet bb1c7a85da844d34be805843bc60d350) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.450351 1168 ts_tablet_manager.cc:1918] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.450470 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 600d711975a74f66b9144758ad0dcfb0
I20250625 13:51:53.451268 590 master_service.cc:948] ReplaceTablet: received request to replace tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.452342 1166 tablet_bootstrap.cc:492] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.452325 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 600d711975a74f66b9144758ad0dcfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.452503 1166 ts_tablet_manager.cc:1397] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.013s
W20250625 13:51:53.452992 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 600d711975a74f66b9144758ad0dcfb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600d711975a74f66b9144758ad0dcfb0 already in progress: creating tablet
I20250625 13:51:53.453344 1166 raft_consensus.cc:357] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.453562 1166 raft_consensus.cc:383] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.453665 1166 raft_consensus.cc:738] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.453905 1166 consensus_queue.cc:260] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.454833 1166 ts_tablet_manager.cc:1428] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:51:53.455137 1166 tablet_bootstrap.cc:492] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.457175 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 600d711975a74f66b9144758ad0dcfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:53.458163 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 600d711975a74f66b9144758ad0dcfb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600d711975a74f66b9144758ad0dcfb0 already in progress: creating tablet
I20250625 13:51:53.458235 1166 tablet_bootstrap.cc:654] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.462216 1170 tablet_bootstrap.cc:492] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.463915 1166 tablet_bootstrap.cc:492] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.464071 1166 ts_tablet_manager.cc:1397] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250625 13:51:53.464708 1166 raft_consensus.cc:357] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.464910 1166 raft_consensus.cc:383] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.464985 1166 raft_consensus.cc:738] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.465217 1166 consensus_queue.cc:260] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.466807 1166 ts_tablet_manager.cc:1428] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:53.467579 1166 tablet_bootstrap.cc:492] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.468045 1170 ts_tablet_manager.cc:1397] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.228s user 0.039s sys 0.080s
I20250625 13:51:53.468858 1170 raft_consensus.cc:357] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.469069 1170 raft_consensus.cc:383] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.469151 1170 raft_consensus.cc:738] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.469195 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 92a90b0dc85844a892ab247ae0d09a12
I20250625 13:51:53.469425 1170 consensus_queue.cc:260] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.469909 1170 ts_tablet_manager.cc:1428] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:51:53.469988 1166 tablet_bootstrap.cc:654] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.470211 1170 tablet_bootstrap.cc:492] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.470448 1168 log.cc:1199] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/c87280e6a3e84e6ba5b0a90e1e7ecec3
I20250625 13:51:53.470888 1168 ts_tablet_manager.cc:1939] T c87280e6a3e84e6ba5b0a90e1e7ecec3 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:53.453411 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 600d711975a74f66b9144758ad0dcfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.471982 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 92a90b0dc85844a892ab247ae0d09a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.472203 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 92a90b0dc85844a892ab247ae0d09a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.472806 1167 tablet_replica.cc:331] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:53.472859 1165 tablet_replica.cc:331] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:53.472986 1167 raft_consensus.cc:2241] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.473104 1165 raft_consensus.cc:2241] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.473169 1167 raft_consensus.cc:2270] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.473320 1165 raft_consensus.cc:2270] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.460232 1171 tablet_bootstrap.cc:492] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.473757 1167 ts_tablet_manager.cc:1905] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.474258 1165 ts_tablet_manager.cc:1905] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.474764 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet c87280e6a3e84e6ba5b0a90e1e7ecec3 (table default.Table_d837899f74444b0a8e38179bb8dea2ee [id=bda1dd723374447b8a351345d7d3ae43]) successfully deleted
I20250625 13:51:53.476009 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_98ae9c7310a24e8d988b919a5ec496e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.476749 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98ae9c7310a24e8d988b919a5ec496e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.478403 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a90cc30efd7145fdbaa7ebdc5db04107"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.478917 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a90cc30efd7145fdbaa7ebdc5db04107 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.473906 1171 ts_tablet_manager.cc:1397] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.170s user 0.009s sys 0.012s
I20250625 13:51:53.471958 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 92a90b0dc85844a892ab247ae0d09a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.471860 1168 tablet_replica.cc:331] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:53.483413 1168 raft_consensus.cc:2241] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.483759 1171 raft_consensus.cc:357] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.483980 1171 raft_consensus.cc:383] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.484068 1171 raft_consensus.cc:738] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.484324 1171 consensus_queue.cc:260] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.483702 1168 raft_consensus.cc:2270] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.484840 1171 ts_tablet_manager.cc:1428] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:53.485854 1171 tablet_bootstrap.cc:492] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.486675 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d524089e6ab044ec961cfc9f587e8f86
I20250625 13:51:53.486819 1168 ts_tablet_manager.cc:1905] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.487913 889 tablet_service.cc:1515] Processing DeleteTablet for tablet d524089e6ab044ec961cfc9f587e8f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.488376 750 tablet_service.cc:1515] Processing DeleteTablet for tablet d524089e6ab044ec961cfc9f587e8f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.491565 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet d524089e6ab044ec961cfc9f587e8f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.492050 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f117921bca85463c82e8673d61c2e9a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:53.492305 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 96d491878b8343be93892aa594ca9da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:53.492619 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f117921bca85463c82e8673d61c2e9a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.494138 1171 tablet_bootstrap.cc:654] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.496037 1166 tablet_bootstrap.cc:492] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.496263 1166 ts_tablet_manager.cc:1397] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20250625 13:51:53.497745 1166 raft_consensus.cc:357] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.498052 1166 raft_consensus.cc:383] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.498221 1166 raft_consensus.cc:738] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.499738 1167 ts_tablet_manager.cc:1918] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.499898 1167 log.cc:1199] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/92a90b0dc85844a892ab247ae0d09a12
I20250625 13:51:53.500278 1167 ts_tablet_manager.cc:1939] T 92a90b0dc85844a892ab247ae0d09a12 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:53.501416 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 92a90b0dc85844a892ab247ae0d09a12 (table default.Table_85c33778738445ae9981ea2f1902e0ec [id=caf5ae6d8d9c482491f4b6d21c421f26]) successfully deleted
I20250625 13:51:53.501480 1167 tablet_replica.cc:331] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:53.501691 1167 raft_consensus.cc:2241] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.501858 1167 raft_consensus.cc:2270] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.501777 1166 consensus_queue.cc:260] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:53.501972 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d524089e6ab044ec961cfc9f587e8f86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d524089e6ab044ec961cfc9f587e8f86 already in progress: creating tablet
I20250625 13:51:53.505167 1166 ts_tablet_manager.cc:1428] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 13:51:53.506757 1171 tablet_bootstrap.cc:492] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.506929 1171 ts_tablet_manager.cc:1397] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250625 13:51:53.507786 1165 ts_tablet_manager.cc:1918] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.507778 1171 raft_consensus.cc:357] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.508040 1171 raft_consensus.cc:383] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.508149 1171 raft_consensus.cc:738] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.508411 1171 consensus_queue.cc:260] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.509001 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cbaa018753d8455199f3869323fb5582
I20250625 13:51:53.509400 593 catalog_manager.cc:6652] ReplaceTablet: tablet cbaa018753d8455199f3869323fb5582 deleted and replaced by tablet cd4f0857f52449e1b26fde09c6c7c9d4
I20250625 13:51:53.510329 889 tablet_service.cc:1515] Processing DeleteTablet for tablet cbaa018753d8455199f3869323fb5582 with delete_type TABLET_DATA_DELETED (replaced by tablet cd4f0857f52449e1b26fde09c6c7c9d4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.510658 1167 ts_tablet_manager.cc:1905] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.510823 750 tablet_service.cc:1515] Processing DeleteTablet for tablet cbaa018753d8455199f3869323fb5582 with delete_type TABLET_DATA_DELETED (replaced by tablet cd4f0857f52449e1b26fde09c6c7c9d4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.510910 1165 log.cc:1199] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/92a90b0dc85844a892ab247ae0d09a12
I20250625 13:51:53.511325 1165 ts_tablet_manager.cc:1939] T 92a90b0dc85844a892ab247ae0d09a12 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:53.512360 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 92a90b0dc85844a892ab247ae0d09a12 (table default.Table_85c33778738445ae9981ea2f1902e0ec [id=caf5ae6d8d9c482491f4b6d21c421f26]) successfully deleted
I20250625 13:51:53.512892 1171 ts_tablet_manager.cc:1428] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:51:53.513159 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet cbaa018753d8455199f3869323fb5582 with delete_type TABLET_DATA_DELETED (replaced by tablet cd4f0857f52449e1b26fde09c6c7c9d4) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:53.514374 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d524089e6ab044ec961cfc9f587e8f86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d524089e6ab044ec961cfc9f587e8f86 already in progress: creating tablet
I20250625 13:51:53.515357 1171 tablet_bootstrap.cc:492] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:51:53.515558 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet cbaa018753d8455199f3869323fb5582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbaa018753d8455199f3869323fb5582 already in progress: creating tablet
I20250625 13:51:53.517649 1166 tablet_bootstrap.cc:492] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.518934 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_77082a1ec23a48adb1f251559dc90002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:53.519634 1171 tablet_bootstrap.cc:654] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:53.519608 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77082a1ec23a48adb1f251559dc90002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.522564 1166 tablet_bootstrap.cc:654] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:53.529822 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c1a98dc40684954a602567cf3476879 consensus state has no leader
I20250625 13:51:53.529950 1167 ts_tablet_manager.cc:1918] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.530211 1170 tablet_bootstrap.cc:654] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:53.530601 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4f19ddc51324cc18242c6081f3f7de5 consensus state has no leader
I20250625 13:51:53.532016 1167 log.cc:1199] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/96d491878b8343be93892aa594ca9da7
I20250625 13:51:53.532460 1167 ts_tablet_manager.cc:1939] T 96d491878b8343be93892aa594ca9da7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:53.533321 1168 ts_tablet_manager.cc:1918] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.533380 588 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a57aa29d10fd4305a3846e1b06e5d2ea
I20250625 13:51:53.533493 1168 log.cc:1199] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/e7906d7c2fb541dcb1ec8e8dcc729c25
I20250625 13:51:53.533592 588 master_service.cc:948] ReplaceTablet: received request to replace tablet cbaa018753d8455199f3869323fb5582 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.533949 1168 ts_tablet_manager.cc:1939] T e7906d7c2fb541dcb1ec8e8dcc729c25 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:53.539749 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ad4e07371e3b42c5b05954f9537fd6b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:53.539949 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 96d491878b8343be93892aa594ca9da7 (table default.Table_e42b0b5035ee4e30a0909702d04b052f [id=a364d50ed707453181496e70e26984e4]) successfully deleted
W20250625 13:51:53.540210 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet cbaa018753d8455199f3869323fb5582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbaa018753d8455199f3869323fb5582 already in progress: creating tablet
W20250625 13:51:53.540359 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad4e07371e3b42c5b05954f9537fd6b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.541513 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_97d93ac47a9f4bc588d79d54df5a1b61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.542001 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97d93ac47a9f4bc588d79d54df5a1b61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.546289 1168 tablet_replica.cc:331] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:53.546362 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet e7906d7c2fb541dcb1ec8e8dcc729c25 (table default.Table_c32eb049fff54229a1db62dc389476aa [id=bb31d8b38dd4479694c08744f757ef7d]) successfully deleted
I20250625 13:51:53.546473 595 master_service.cc:948] ReplaceTablet: received request to replace tablet a179c3698d424ec7b6d87bf65cb858cc from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.546615 1168 raft_consensus.cc:2241] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.546836 1168 raft_consensus.cc:2270] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.547283 1170 tablet_bootstrap.cc:492] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.547483 1170 ts_tablet_manager.cc:1397] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.003s
I20250625 13:51:53.547469 1168 ts_tablet_manager.cc:1905] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.548305 1170 raft_consensus.cc:357] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.548497 1170 raft_consensus.cc:383] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.548600 1170 raft_consensus.cc:738] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.548859 1170 consensus_queue.cc:260] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.549415 1170 ts_tablet_manager.cc:1428] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:51:53.549731 1170 tablet_bootstrap.cc:492] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.551524 1170 tablet_bootstrap.cc:654] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.554054 1166 tablet_bootstrap.cc:492] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.554721 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.555277 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.556005 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet b44c85b12fe64fc99fcf0a79a5909c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.556674 1167 tablet_replica.cc:331] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:53.557009 1167 raft_consensus.cc:2241] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.558056 1167 raft_consensus.cc:2270] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.558389 1166 ts_tablet_manager.cc:1397] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.002s
I20250625 13:51:53.558432 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c7d9dab52e4c4c3bbd399e6dbc197775"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.558957 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.563872 1167 ts_tablet_manager.cc:1905] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.564539 1166 raft_consensus.cc:357] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.564769 1166 raft_consensus.cc:383] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.564867 1166 raft_consensus.cc:738] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.565196 1166 consensus_queue.cc:260] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.565724 1166 ts_tablet_manager.cc:1428] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:51:53.566080 1166 tablet_bootstrap.cc:492] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.567996 1166 tablet_bootstrap.cc:654] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.570734 1171 tablet_bootstrap.cc:492] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.570917 1171 ts_tablet_manager.cc:1397] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.001s
I20250625 13:51:53.571789 1171 raft_consensus.cc:357] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.572000 1171 raft_consensus.cc:383] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.572054 1050 raft_consensus.cc:3058] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.572091 1171 raft_consensus.cc:738] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.572360 1171 consensus_queue.cc:260] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.572563 1167 ts_tablet_manager.cc:1918] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.572815 1167 log.cc:1199] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/b44c85b12fe64fc99fcf0a79a5909c6c
I20250625 13:51:53.573158 1171 ts_tablet_manager.cc:1428] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:53.573244 1167 ts_tablet_manager.cc:1939] T b44c85b12fe64fc99fcf0a79a5909c6c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:53.573450 1171 tablet_bootstrap.cc:492] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.574445 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet b44c85b12fe64fc99fcf0a79a5909c6c (table default.Table_3f5b9820585e4ed89aa7d0abca37a52c [id=d7e8d11b88e240b89d40754b67570894]) successfully deleted
I20250625 13:51:53.575084 1171 tablet_bootstrap.cc:654] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.575234 1050 raft_consensus.cc:1273] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Refusing update from remote peer 5c1aa07ab6bc4b0089e46951240a37c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:51:53.576380 915 raft_consensus.cc:1273] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Refusing update from remote peer 5c1aa07ab6bc4b0089e46951240a37c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:51:53.577364 1220 consensus_queue.cc:1035] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:53.582886 1220 consensus_queue.cc:1035] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:53.584741 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_84902149ef0541b2956407332a512a7a" } new_table_name: "default.Table_8fc2c990732d4e658580c2bc2cf67f4d" modify_external_catalogs: true
I20250625 13:51:53.594946 1170 tablet_bootstrap.cc:492] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.598156 1168 ts_tablet_manager.cc:1918] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.598338 1168 log.cc:1199] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/b44c85b12fe64fc99fcf0a79a5909c6c
I20250625 13:51:53.595197 1170 ts_tablet_manager.cc:1397] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.010s
I20250625 13:51:53.598783 1168 ts_tablet_manager.cc:1939] T b44c85b12fe64fc99fcf0a79a5909c6c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:53.599452 1170 raft_consensus.cc:357] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.599663 1170 raft_consensus.cc:383] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.599730 1170 raft_consensus.cc:738] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.599803 1168 tablet_replica.cc:331] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:53.599968 1170 consensus_queue.cc:260] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.600031 1168 raft_consensus.cc:2241] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.600196 1168 raft_consensus.cc:2270] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.600229 590 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_a0708b1a41774b1c8ecdf97ea7e66531 [id=c2e58d3856d7401ca7b97d9cce92bbce]). Expected version 4 got 0
I20250625 13:51:53.600416 1170 ts_tablet_manager.cc:1428] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:51:53.600754 1170 tablet_bootstrap.cc:492] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.600800 1168 ts_tablet_manager.cc:1905] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.602903 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_a0708b1a41774b1c8ecdf97ea7e66531 [id=c2e58d3856d7401ca7b97d9cce92bbce]). Expected version 4 got 0
I20250625 13:51:53.602967 1170 tablet_bootstrap.cc:654] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:53.603734 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0723e91650174f5badd74f7756c5a54e consensus state has no leader
I20250625 13:51:53.604913 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 583ee2f2c66841939c93104efc36facb (DEFAULT_TABLE table=default.Table_5964baca5b4a4701bf5a1b86d494b4e0 [id=c12c680f4ece4270ba65d0e2dc275254]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.605412 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583ee2f2c66841939c93104efc36facb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.607576 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 799f2edd960d45c1ace9195283a042e2 (DEFAULT_TABLE table=default.Table_eb08aa18af9641049c9d7345375fa250 [id=82115aa29a8f456b8d7c2f3d9caa17e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.607928 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799f2edd960d45c1ace9195283a042e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.609377 1166 tablet_bootstrap.cc:492] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.609556 1166 ts_tablet_manager.cc:1397] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.005s
I20250625 13:51:53.610527 1166 raft_consensus.cc:357] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.610734 1166 raft_consensus.cc:383] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.610847 1166 raft_consensus.cc:738] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.611116 1166 consensus_queue.cc:260] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.611631 1027 tablet_service.cc:1468] Processing CreateTablet for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 (DEFAULT_TABLE table=default.Table_4212481baebd45429d103db60e844de5 [id=6e50ba5e257842e080a70caa1679952d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.611776 1166 ts_tablet_manager.cc:1428] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250625 13:51:53.612030 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bb7ebc40a14ecaa0c692a8e1117db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.612134 1166 tablet_bootstrap.cc:492] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.612684 1171 tablet_bootstrap.cc:492] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.612820 1171 ts_tablet_manager.cc:1397] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20250625 13:51:53.613567 1171 raft_consensus.cc:357] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.613744 1171 raft_consensus.cc:383] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.613818 1171 raft_consensus.cc:738] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.614097 1166 tablet_bootstrap.cc:654] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.614492 1171 consensus_queue.cc:260] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.615738 1171 ts_tablet_manager.cc:1428] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250625 13:51:53.616185 1171 tablet_bootstrap.cc:492] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.618328 889 tablet_service.cc:1468] Processing CreateTablet for tablet 583ee2f2c66841939c93104efc36facb (DEFAULT_TABLE table=default.Table_5964baca5b4a4701bf5a1b86d494b4e0 [id=c12c680f4ece4270ba65d0e2dc275254]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.618716 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583ee2f2c66841939c93104efc36facb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.622539 1168 ts_tablet_manager.cc:1918] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.622702 1168 log.cc:1199] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/96d491878b8343be93892aa594ca9da7
I20250625 13:51:53.623121 1168 ts_tablet_manager.cc:1939] T 96d491878b8343be93892aa594ca9da7 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:53.623296 893 tablet_service.cc:1468] Processing CreateTablet for tablet 799f2edd960d45c1ace9195283a042e2 (DEFAULT_TABLE table=default.Table_eb08aa18af9641049c9d7345375fa250 [id=82115aa29a8f456b8d7c2f3d9caa17e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.624154 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet b44c85b12fe64fc99fcf0a79a5909c6c (table default.Table_3f5b9820585e4ed89aa7d0abca37a52c [id=d7e8d11b88e240b89d40754b67570894]) successfully deleted
I20250625 13:51:53.624872 1170 tablet_bootstrap.cc:492] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.625043 1170 ts_tablet_manager.cc:1397] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250625 13:51:53.626264 750 tablet_service.cc:1468] Processing CreateTablet for tablet 583ee2f2c66841939c93104efc36facb (DEFAULT_TABLE table=default.Table_5964baca5b4a4701bf5a1b86d494b4e0 [id=c12c680f4ece4270ba65d0e2dc275254]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.626737 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 583ee2f2c66841939c93104efc36facb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.627440 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 96d491878b8343be93892aa594ca9da7 (table default.Table_e42b0b5035ee4e30a0909702d04b052f [id=a364d50ed707453181496e70e26984e4]) successfully deleted
I20250625 13:51:53.629029 1171 tablet_bootstrap.cc:654] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.628953 1170 raft_consensus.cc:357] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.629145 1170 raft_consensus.cc:383] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.629221 1170 raft_consensus.cc:738] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.629846 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a17f862a8a584047a8f0220f2eced869 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:51 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.631640 753 tablet_service.cc:1468] Processing CreateTablet for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 (DEFAULT_TABLE table=default.Table_4212481baebd45429d103db60e844de5 [id=6e50ba5e257842e080a70caa1679952d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.632023 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bb7ebc40a14ecaa0c692a8e1117db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.635417 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9ba63184ca2b4badafcf08309ef75164 (DEFAULT_TABLE table=default.Table_ac6eba07f2c94c75a0be2404ab15f0d9 [id=1cd7ba4de230481e8b0c57c1c585fbd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.635540 1170 consensus_queue.cc:260] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.635864 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ba63184ca2b4badafcf08309ef75164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.636153 1170 ts_tablet_manager.cc:1428] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 13:51:53.636502 1170 tablet_bootstrap.cc:492] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.636521 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9ba63184ca2b4badafcf08309ef75164 (DEFAULT_TABLE table=default.Table_ac6eba07f2c94c75a0be2404ab15f0d9 [id=1cd7ba4de230481e8b0c57c1c585fbd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.636987 896 tablet_service.cc:1468] Processing CreateTablet for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 (DEFAULT_TABLE table=default.Table_0ae4786bc45842eea9376ff1f404d621 [id=13d38d05aad34e22884b9e07610b0e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.637686 894 tablet_service.cc:1468] Processing CreateTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 (DEFAULT_TABLE table=default.Table_84902149ef0541b2956407332a512a7a [id=75b34bf006e040d18266d0c90986afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.638530 1170 tablet_bootstrap.cc:654] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.638913 883 tablet_service.cc:1468] Processing CreateTablet for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 (DEFAULT_TABLE table=default.Table_4212481baebd45429d103db60e844de5 [id=6e50ba5e257842e080a70caa1679952d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.639712 756 tablet_service.cc:1468] Processing CreateTablet for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 (DEFAULT_TABLE table=default.Table_0ae4786bc45842eea9376ff1f404d621 [id=13d38d05aad34e22884b9e07610b0e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.640098 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a7862601f04fd7a938f7e5d5a4c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.640437 1166 tablet_bootstrap.cc:492] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.640621 1166 ts_tablet_manager.cc:1397] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250625 13:51:53.641484 1166 raft_consensus.cc:357] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.641695 1166 raft_consensus.cc:383] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.641929 1166 raft_consensus.cc:738] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.642204 1166 consensus_queue.cc:260] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.643157 1166 ts_tablet_manager.cc:1428] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:53.643136 884 tablet_service.cc:1468] Processing CreateTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a (DEFAULT_TABLE table=default.Table_7ccc261a6367449583a5c30495bfe2a9 [id=5436e1b638d14b00a66842a43cc5c085]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.643615 1166 tablet_bootstrap.cc:492] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:53.643579 751 tablet_service.cc:1468] Processing CreateTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 (DEFAULT_TABLE table=default.Table_84902149ef0541b2956407332a512a7a [id=75b34bf006e040d18266d0c90986afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.643971 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c72f9ae92a4537979fb479eb4467c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.645628 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 (DEFAULT_TABLE table=default.Table_0ae4786bc45842eea9376ff1f404d621 [id=13d38d05aad34e22884b9e07610b0e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.639293 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 9ba63184ca2b4badafcf08309ef75164 (DEFAULT_TABLE table=default.Table_ac6eba07f2c94c75a0be2404ab15f0d9 [id=1cd7ba4de230481e8b0c57c1c585fbd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.647575 752 tablet_service.cc:1468] Processing CreateTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d (DEFAULT_TABLE table=default.Table_1be3d9db61ab4b56b98b70af09df98b3 [id=68e44a6915224107b42ca15194e2fed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.647629 747 tablet_service.cc:1468] Processing CreateTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a (DEFAULT_TABLE table=default.Table_7ccc261a6367449583a5c30495bfe2a9 [id=5436e1b638d14b00a66842a43cc5c085]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.648108 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f449c80b604a6498f6fd1d1ba4770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.648586 758 tablet_service.cc:1468] Processing CreateTablet for tablet 799f2edd960d45c1ace9195283a042e2 (DEFAULT_TABLE table=default.Table_eb08aa18af9641049c9d7345375fa250 [id=82115aa29a8f456b8d7c2f3d9caa17e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.648991 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799f2edd960d45c1ace9195283a042e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.648103 1166 tablet_bootstrap.cc:654] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.651371 1291 tablet.cc:1722] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:51:53.651590 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.658041 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a (DEFAULT_TABLE table=default.Table_7ccc261a6367449583a5c30495bfe2a9 [id=5436e1b638d14b00a66842a43cc5c085]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.658460 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 (DEFAULT_TABLE table=default.Table_84902149ef0541b2956407332a512a7a [id=75b34bf006e040d18266d0c90986afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.665639 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a7862601f04fd7a938f7e5d5a4c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.666386 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c72f9ae92a4537979fb479eb4467c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.666602 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ba63184ca2b4badafcf08309ef75164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.667152 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f449c80b604a6498f6fd1d1ba4770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.674047 1258 mvcc.cc:204] Tried to move back new op lower bound from 7171520567550836736 to 7171520564846710784. Current Snapshot: MvccSnapshot[applied={T|T < 7171520567550836736}]
I20250625 13:51:53.674619 1266 tablet.cc:1722] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:51:53.676072 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ba63184ca2b4badafcf08309ef75164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.676968 1171 tablet_bootstrap.cc:492] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.677150 1171 ts_tablet_manager.cc:1397] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.006s
I20250625 13:51:53.678737 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5964baca5b4a4701bf5a1b86d494b4e0" } modify_external_catalogs: true
I20250625 13:51:53.679031 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5964baca5b4a4701bf5a1b86d494b4e0" } modify_external_catalogs: true
I20250625 13:51:53.679055 1171 raft_consensus.cc:357] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.679320 1171 raft_consensus.cc:383] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.679407 1171 raft_consensus.cc:738] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.679540 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_bcac7658498546e39e2b1f2d30910433 [id=c2e58d3856d7401ca7b97d9cce92bbce]). Expected version 5 got 0
I20250625 13:51:53.679767 1171 consensus_queue.cc:260] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.681340 592 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
I20250625 13:51:53.681452 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b079a0f225cf4ac98c2efc03bb74b889" } modify_external_catalogs: true
I20250625 13:51:53.681699 1171 ts_tablet_manager.cc:1428] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250625 13:51:53.681653 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b079a0f225cf4ac98c2efc03bb74b889" } modify_external_catalogs: true
I20250625 13:51:53.682480 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f117921bca85463c82e8673d61c2e9a6" } new_table_name: "default.Table_c1e172984ca643a9b7c88d09f4d5221a" modify_external_catalogs: true
I20250625 13:51:53.683810 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d675ca14ac884a35aab70e66cdcef846"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:53.684016 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_efbacb69562d4fe484a0cafa47e8985e" } modify_external_catalogs: true
I20250625 13:51:53.684177 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_efbacb69562d4fe484a0cafa47e8985e" } modify_external_catalogs: true
I20250625 13:51:53.684296 1171 tablet_bootstrap.cc:492] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:51:53.684391 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d675ca14ac884a35aab70e66cdcef846 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.686682 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799f2edd960d45c1ace9195283a042e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.688166 1171 tablet_bootstrap.cc:654] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.694911 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a7862601f04fd7a938f7e5d5a4c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.699540 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c72f9ae92a4537979fb479eb4467c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.701774 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d (DEFAULT_TABLE table=default.Table_1be3d9db61ab4b56b98b70af09df98b3 [id=68e44a6915224107b42ca15194e2fed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.702185 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.703928 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9bb7ebc40a14ecaa0c692a8e1117db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.707564 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f449c80b604a6498f6fd1d1ba4770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.712577 886 tablet_service.cc:1468] Processing CreateTablet for tablet 50786a4c25274ffb8f9910c5dafe3933 (DEFAULT_TABLE table=default.Table_e930f596e0f6454c8af779d2e0f7ff76 [id=d7cbe338b2554a6e99b07325632aaa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.713107 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50786a4c25274ffb8f9910c5dafe3933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.713614 895 tablet_service.cc:1468] Processing CreateTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 (DEFAULT_TABLE table=default.Table_9e86330e77034242a9aa4fbd274c3151 [id=0b06b619484e459f949d8acb2bb19512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.714170 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73f420a4d7a438eadde3d6af64f0a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.712602 889 tablet_service.cc:1468] Processing CreateTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d (DEFAULT_TABLE table=default.Table_1be3d9db61ab4b56b98b70af09df98b3 [id=68e44a6915224107b42ca15194e2fed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.715863 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.740358 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 (DEFAULT_TABLE table=default.Table_9e86330e77034242a9aa4fbd274c3151 [id=0b06b619484e459f949d8acb2bb19512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.741641 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73f420a4d7a438eadde3d6af64f0a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.743696 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 50786a4c25274ffb8f9910c5dafe3933 (DEFAULT_TABLE table=default.Table_e930f596e0f6454c8af779d2e0f7ff76 [id=d7cbe338b2554a6e99b07325632aaa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.744824 1168 tablet_replica.cc:331] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:53.745101 1168 raft_consensus.cc:2241] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.745371 1168 raft_consensus.cc:2270] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.746011 1168 ts_tablet_manager.cc:1905] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.757102 1168 ts_tablet_manager.cc:1918] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.757881 1168 log.cc:1199] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/3dbeb984fadf4a0ebc50087c1a921f07
I20250625 13:51:53.758432 1168 ts_tablet_manager.cc:1939] T 3dbeb984fadf4a0ebc50087c1a921f07 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:53.758458 1171 tablet_bootstrap.cc:492] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.758677 1171 ts_tablet_manager.cc:1397] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.077s user 0.020s sys 0.014s
I20250625 13:51:53.759768 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50786a4c25274ffb8f9910c5dafe3933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.759768 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 3dbeb984fadf4a0ebc50087c1a921f07 (table default.Table_9875a118a9a74b7486cc83256370ddfd [id=72787a6701a64f08b90a8954cca4f049]) successfully deleted
I20250625 13:51:53.760192 1168 tablet_replica.cc:331] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:53.760470 1168 raft_consensus.cc:2241] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.760778 1168 raft_consensus.cc:2270] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.761159 1171 raft_consensus.cc:357] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.761348 1171 raft_consensus.cc:383] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.761427 1171 raft_consensus.cc:738] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.761631 1171 consensus_queue.cc:260] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.762673 1171 ts_tablet_manager.cc:1428] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:51:53.763067 1171 tablet_bootstrap.cc:492] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.763796 1168 ts_tablet_manager.cc:1905] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.765787 1171 tablet_bootstrap.cc:654] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.770753 1168 ts_tablet_manager.cc:1918] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.770963 1168 log.cc:1199] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/3c282591e90c40948563fda06453e4d4
I20250625 13:51:53.771394 1168 ts_tablet_manager.cc:1939] T 3c282591e90c40948563fda06453e4d4 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:53.772586 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 3c282591e90c40948563fda06453e4d4 (table default.Table_893b774d7d574f8e89a027212ab17b74 [id=18ba0d9199d14289bed69ea544d3dced]) successfully deleted
W20250625 13:51:53.773368 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bf54376db991424b8de0e8f252fcb43f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf54376db991424b8de0e8f252fcb43f already in progress: creating tablet
W20250625 13:51:53.774055 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet da4f9a46a5804da3aff1c9a87b2e4984 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da4f9a46a5804da3aff1c9a87b2e4984 already in progress: creating tablet
I20250625 13:51:53.774243 1171 tablet_bootstrap.cc:492] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.774441 1171 ts_tablet_manager.cc:1397] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
W20250625 13:51:53.774796 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1a7b72c2c464421eacfcf64008c3aea8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a7b72c2c464421eacfcf64008c3aea8 already in progress: creating tablet
I20250625 13:51:53.775373 1171 raft_consensus.cc:357] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.779345 1220 raft_consensus.cc:491] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.784539 1170 tablet_bootstrap.cc:492] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.784719 1170 ts_tablet_manager.cc:1397] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.148s user 0.014s sys 0.026s
I20250625 13:51:53.788384 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.790970 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b13c5d5237f044c9bad038533aecc142" } modify_external_catalogs: true
I20250625 13:51:53.791174 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b13c5d5237f044c9bad038533aecc142" } modify_external_catalogs: true
I20250625 13:51:53.791676 1170 raft_consensus.cc:357] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.791975 1170 raft_consensus.cc:383] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.792057 1170 raft_consensus.cc:738] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.792369 1170 consensus_queue.cc:260] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.793136 1170 ts_tablet_manager.cc:1428] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:51:53.793576 1170 tablet_bootstrap.cc:492] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.795519 1170 tablet_bootstrap.cc:654] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.796007 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 36da1da539e24f55978d516873cb0b27
I20250625 13:51:53.797441 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a179c3698d424ec7b6d87bf65cb858cc
I20250625 13:51:53.798431 595 catalog_manager.cc:6652] ReplaceTablet: tablet a179c3698d424ec7b6d87bf65cb858cc deleted and replaced by tablet 0197f37545604f59b582ccbaa6a1a853
I20250625 13:51:53.798769 595 master_service.cc:948] ReplaceTablet: received request to replace tablet ce84f74470ad428abb2113efa170ca24 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:53.799264 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04479dfb34584f0891c604ef72bf93b5 has no consensus state
I20250625 13:51:53.801239 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
W20250625 13:51:53.798446 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85c72f9ae92a4537979fb479eb4467c4 consensus state has no leader
I20250625 13:51:53.802883 1165 tablet_replica.cc:331] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:53.802983 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 583ee2f2c66841939c93104efc36facb
I20250625 13:51:53.803540 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 0723e91650174f5badd74f7756c5a54e from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.804736 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 36da1da539e24f55978d516873cb0b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.806183 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.807044 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 583ee2f2c66841939c93104efc36facb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.807036 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 583ee2f2c66841939c93104efc36facb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.807806 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a179c3698d424ec7b6d87bf65cb858cc with delete_type TABLET_DATA_DELETED (replaced by tablet 0197f37545604f59b582ccbaa6a1a853) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.809412 1168 tablet_replica.cc:331] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:53.809442 1171 raft_consensus.cc:383] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.809571 1171 raft_consensus.cc:738] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.809863 884 tablet_service.cc:1515] Processing DeleteTablet for tablet a179c3698d424ec7b6d87bf65cb858cc with delete_type TABLET_DATA_DELETED (replaced by tablet 0197f37545604f59b582ccbaa6a1a853) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.810134 1170 tablet_bootstrap.cc:492] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:53.810123 891 tablet_service.cc:1515] Processing DeleteTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b549bc8cead4af4a72f41f51655602c) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.810276 1170 ts_tablet_manager.cc:1397] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20250625 13:51:53.810078 1168 raft_consensus.cc:2241] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.810304 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 36da1da539e24f55978d516873cb0b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:53.810145 1171 consensus_queue.cc:260] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.810946 1171 ts_tablet_manager.cc:1428] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20250625 13:51:53.811110 1168 raft_consensus.cc:2270] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.811106 1170 raft_consensus.cc:357] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.811283 1170 raft_consensus.cc:383] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.811388 1170 raft_consensus.cc:738] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.811597 1170 consensus_queue.cc:260] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.812064 1170 ts_tablet_manager.cc:1428] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:51:53.812377 1170 tablet_bootstrap.cc:492] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:53.812176 1220 raft_consensus.cc:513] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.812676 1171 tablet_bootstrap.cc:492] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.809625 756 tablet_service.cc:1515] Processing DeleteTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b549bc8cead4af4a72f41f51655602c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.814033 1170 tablet_bootstrap.cc:654] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.814325 1171 tablet_bootstrap.cc:654] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.815263 753 tablet_service.cc:1515] Processing DeleteTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5ba76c130a4711af5b21e9eb8562e7) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:53.807964 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 583ee2f2c66841939c93104efc36facb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 583ee2f2c66841939c93104efc36facb already in progress: creating tablet
W20250625 13:51:53.818940 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a179c3698d424ec7b6d87bf65cb858cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a179c3698d424ec7b6d87bf65cb858cc already in progress: creating tablet
W20250625 13:51:53.819175 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 36da1da539e24f55978d516873cb0b27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36da1da539e24f55978d516873cb0b27 already in progress: creating tablet
I20250625 13:51:53.820549 754 tablet_service.cc:1468] Processing CreateTablet for tablet 50786a4c25274ffb8f9910c5dafe3933 (DEFAULT_TABLE table=default.Table_e930f596e0f6454c8af779d2e0f7ff76 [id=d7cbe338b2554a6e99b07325632aaa33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.821029 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50786a4c25274ffb8f9910c5dafe3933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.822538 1165 raft_consensus.cc:2241] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.824772 1166 tablet_bootstrap.cc:492] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:53.824940 1166 ts_tablet_manager.cc:1397] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.182s user 0.012s sys 0.029s
I20250625 13:51:53.828645 1165 raft_consensus.cc:2270] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:53.828600 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0723e91650174f5badd74f7756c5a54e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:53.828809 1047 raft_consensus.cc:2466] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:53.830925 594 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 49333f456790418fbc3b72161dacca8a
I20250625 13:51:53.831902 1168 ts_tablet_manager.cc:1905] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.834498 711 leader_election.cc:304] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:53.835294 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 36da1da539e24f55978d516873cb0b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.835835 1220 leader_election.cc:290] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.838696 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 921163dc5f454c9a97e72b005f9dba19
I20250625 13:51:53.840062 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 87327f74beb845768bbfd5ae6d2bcf88 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.840473 758 tablet_service.cc:1468] Processing CreateTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 (DEFAULT_TABLE table=default.Table_9e86330e77034242a9aa4fbd274c3151 [id=0b06b619484e459f949d8acb2bb19512]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:53.840950 1165 ts_tablet_manager.cc:1905] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:53.842283 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 921163dc5f454c9a97e72b005f9dba19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.842425 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 921163dc5f454c9a97e72b005f9dba19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.842671 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 583ee2f2c66841939c93104efc36facb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:53.844502 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 921163dc5f454c9a97e72b005f9dba19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 921163dc5f454c9a97e72b005f9dba19 already in progress: creating tablet
I20250625 13:51:53.849084 1171 tablet_bootstrap.cc:492] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.849045 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a1d1eec1b64d49d9a7c378dc4772a5db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:53.849249 1171 ts_tablet_manager.cc:1397] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.008s
W20250625 13:51:53.849762 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1d1eec1b64d49d9a7c378dc4772a5db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.850055 1171 raft_consensus.cc:357] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.850248 1171 raft_consensus.cc:383] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.850351 1171 raft_consensus.cc:738] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.850610 1171 consensus_queue.cc:260] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.852213 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4755ed4ecc244e44a38ad91f159a119b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:53.852641 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_97d93ac47a9f4bc588d79d54df5a1b61" } new_table_name: "default.Table_72b53bc133ea42568572d075e5493c2e" modify_external_catalogs: true
W20250625 13:51:53.852841 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4755ed4ecc244e44a38ad91f159a119b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.856951 1171 ts_tablet_manager.cc:1428] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:51:53.857256 1171 tablet_bootstrap.cc:492] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.858999 1171 tablet_bootstrap.cc:654] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.859917 751 tablet_service.cc:1515] Processing DeleteTablet for tablet a179c3698d424ec7b6d87bf65cb858cc with delete_type TABLET_DATA_DELETED (replaced by tablet 0197f37545604f59b582ccbaa6a1a853) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.860198 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73f420a4d7a438eadde3d6af64f0a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:53.862192 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bcac7658498546e39e2b1f2d30910433" } new_table_name: "default.Table_e5291e5468c04390947d9ea217535cf6" modify_external_catalogs: true
I20250625 13:51:53.867442 1166 raft_consensus.cc:357] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.867699 1166 raft_consensus.cc:383] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.868359 1166 raft_consensus.cc:738] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.868589 1166 consensus_queue.cc:260] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.868952 1171 tablet_bootstrap.cc:492] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.869117 1171 ts_tablet_manager.cc:1397] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250625 13:51:53.870014 1171 raft_consensus.cc:357] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:53.870231 1171 raft_consensus.cc:383] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.870311 1171 raft_consensus.cc:738] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.870561 1171 consensus_queue.cc:260] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:53.872148 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_05f514ab26624c719af36996404ef419" } modify_external_catalogs: true
I20250625 13:51:53.872484 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_05f514ab26624c719af36996404ef419" } modify_external_catalogs: true
I20250625 13:51:53.873275 1263 raft_consensus.cc:491] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.873950 1171 ts_tablet_manager.cc:1428] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250625 13:51:53.879830 1171 tablet_bootstrap.cc:492] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:53.881829 1297 raft_consensus.cc:491] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.882130 1297 raft_consensus.cc:513] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.883159 1297 leader_election.cc:290] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.883452 1297 raft_consensus.cc:491] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.883690 1297 raft_consensus.cc:513] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.884397 1297 leader_election.cc:290] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.884650 1297 raft_consensus.cc:2802] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:53.884804 1297 raft_consensus.cc:491] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:53.884994 1297 raft_consensus.cc:3058] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.904636 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561] alter complete (version 1)
I20250625 13:51:53.911082 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0191a529552e4cb3b2ed7b47c274b71b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.912606 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0191a529552e4cb3b2ed7b47c274b71b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.885627 1220 raft_consensus.cc:491] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.916575 1220 raft_consensus.cc:513] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.917479 1220 leader_election.cc:290] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.920847 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 87327f74beb845768bbfd5ae6d2bcf88
I20250625 13:51:53.921499 591 catalog_manager.cc:6652] ReplaceTablet: tablet 87327f74beb845768bbfd5ae6d2bcf88 deleted and replaced by tablet f00e72e103f443e886f0df690cff4b6f
I20250625 13:51:53.923309 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ce84f74470ad428abb2113efa170ca24
I20250625 13:51:53.923759 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 with delete_type TABLET_DATA_DELETED (replaced by tablet f00e72e103f443e886f0df690cff4b6f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:53.923956 595 catalog_manager.cc:6652] ReplaceTablet: tablet ce84f74470ad428abb2113efa170ca24 deleted and replaced by tablet 01ba67b500b646d7aa55315f7ee23ed6
I20250625 13:51:53.925463 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet ce84f74470ad428abb2113efa170ca24 with delete_type TABLET_DATA_DELETED (replaced by tablet 01ba67b500b646d7aa55315f7ee23ed6) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:53.926285 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 87327f74beb845768bbfd5ae6d2bcf88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87327f74beb845768bbfd5ae6d2bcf88 already in progress: creating tablet
W20250625 13:51:53.926647 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ce84f74470ad428abb2113efa170ca24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce84f74470ad428abb2113efa170ca24 already in progress: creating tablet
I20250625 13:51:53.927284 592 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 0197f37545604f59b582ccbaa6a1a853
W20250625 13:51:53.929010 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15e58fbec94e4ff9b7ac9830b576fe7f has no consensus state
I20250625 13:51:53.931036 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ce84f74470ad428abb2113efa170ca24 with delete_type TABLET_DATA_DELETED (replaced by tablet 01ba67b500b646d7aa55315f7ee23ed6) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.931526 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 with delete_type TABLET_DATA_DELETED (replaced by tablet f00e72e103f443e886f0df690cff4b6f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.932497 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0723e91650174f5badd74f7756c5a54e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:53.932729 915 raft_consensus.cc:2466] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:53.934199 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bc823b8c0cbd4aa8a4c7465e9e36410a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.934731 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc823b8c0cbd4aa8a4c7465e9e36410a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.938346 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8c4176baf42b43658ec245c90d86cb98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.939354 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c4176baf42b43658ec245c90d86cb98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.941004 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_59ed24a1a856411ba6d9c5504c4c9a7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.941741 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59ed24a1a856411ba6d9c5504c4c9a7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.943653 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9798c064000341b5bc384b0dee1787d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.944406 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9798c064000341b5bc384b0dee1787d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.945720 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_eb08aa18af9641049c9d7345375fa250" } new_table_name: "default.Table_12b254975e4a431ba57506a99aed12b4" modify_external_catalogs: true
I20250625 13:51:53.949823 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 799f2edd960d45c1ace9195283a042e2 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:53.873422 1263 raft_consensus.cc:513] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.951722 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 15750915ff7746ffa66509184be55dc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e0f2fa5d5614ad394e6e12ec62814d4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.952710 1263 leader_election.cc:290] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.952872 1263 raft_consensus.cc:491] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.953013 1263 raft_consensus.cc:513] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.953550 1263 leader_election.cc:290] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.953704 1263 raft_consensus.cc:491] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.953805 1263 raft_consensus.cc:513] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.954408 1263 leader_election.cc:290] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.955214 1171 tablet_bootstrap.cc:654] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:53.957420 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca5dda8b27b34dafb20d8ed8afc58460" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:53.958331 849 leader_election.cc:343] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:53.958961 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da4f9a46a5804da3aff1c9a87b2e4984" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:53.959328 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a179c3698d424ec7b6d87bf65cb858cc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:53.959533 1047 raft_consensus.cc:2466] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:51:53.960459 849 leader_election.cc:343] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:53.960847 849 leader_election.cc:304] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:53.961199 1263 raft_consensus.cc:2802] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:53.961321 1263 raft_consensus.cc:491] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:53.961414 1263 raft_consensus.cc:3058] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.902431 1165 ts_tablet_manager.cc:1918] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:53.962007 1165 log.cc:1199] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/872724cd95bf473cb40e5f8ed8b5bd55
I20250625 13:51:53.962371 1165 ts_tablet_manager.cc:1939] T 872724cd95bf473cb40e5f8ed8b5bd55 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:53.963546 1165 tablet_replica.cc:331] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:53.963726 1165 raft_consensus.cc:2241] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:53.915349 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:53.963917 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca5dda8b27b34dafb20d8ed8afc58460" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:53.964408 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 872724cd95bf473cb40e5f8ed8b5bd55 (table default.Table_c94d10947ecf49fba6f7608086ef35bc [id=263e68a7d9344016afdb8f4dc5b27b4f]) successfully deleted
I20250625 13:51:53.964417 1263 raft_consensus.cc:513] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.964789 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a179c3698d424ec7b6d87bf65cb858cc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:53.965389 851 leader_election.cc:343] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:53.965286 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da4f9a46a5804da3aff1c9a87b2e4984" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:53.965595 776 raft_consensus.cc:2466] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:53.966740 1263 leader_election.cc:290] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:53.967343 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a179c3698d424ec7b6d87bf65cb858cc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:53.967566 1047 raft_consensus.cc:3058] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:53.967604 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 138314us
I20250625 13:51:53.968228 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da4f9a46a5804da3aff1c9a87b2e4984" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:53.969251 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40a6375e98a04f95aa2bbde6b9b0228f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:53.969432 1050 raft_consensus.cc:2466] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:53.970619 1243 mvcc.cc:204] Tried to move back new op lower bound from 7171520567550836736 to 7171520564846710784. Current Snapshot: MvccSnapshot[applied={T|T < 7171520567550836736}]
I20250625 13:51:53.971426 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a179c3698d424ec7b6d87bf65cb858cc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:53.971740 1047 raft_consensus.cc:2466] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:53.972172 849 leader_election.cc:304] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:53.972290 851 leader_election.cc:304] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:53.972725 1263 raft_consensus.cc:2802] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:51:53.972824 851 leader_election.cc:343] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:53.972944 1263 raft_consensus.cc:695] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:53.973664 1263 consensus_queue.cc:237] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.974962 1174 raft_consensus.cc:491] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:53.975142 1174 raft_consensus.cc:513] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.975414 1307 raft_consensus.cc:2802] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:53.975529 1307 raft_consensus.cc:491] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:53.975683 1307 raft_consensus.cc:3058] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.975769 1174 leader_election.cc:290] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:53.975945 1171 tablet_bootstrap.cc:492] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:53.976109 1171 ts_tablet_manager.cc:1397] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.001s
I20250625 13:51:53.977038 1171 raft_consensus.cc:357] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:53.977221 1171 raft_consensus.cc:383] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:53.977329 1171 raft_consensus.cc:738] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:53.977218 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "872724cd95bf473cb40e5f8ed8b5bd55" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:53.977563 915 raft_consensus.cc:2466] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:53.977509 1171 consensus_queue.cc:260] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:53.978157 983 leader_election.cc:304] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:53.978518 1174 raft_consensus.cc:2802] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:53.978627 1174 raft_consensus.cc:491] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:53.978757 1174 raft_consensus.cc:3058] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:53.981798 1174 raft_consensus.cc:513] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.982800 1174 leader_election.cc:290] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:53.983894 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a21873ae3bbd4a1aaf287e80fe14c12e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.984696 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a21873ae3bbd4a1aaf287e80fe14c12e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:53.985574 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "872724cd95bf473cb40e5f8ed8b5bd55" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:53.986397 915 raft_consensus.cc:3058] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:53.987681 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 799f2edd960d45c1ace9195283a042e2 consensus state has no leader
I20250625 13:51:53.990130 915 raft_consensus.cc:2466] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:53.990773 983 leader_election.cc:304] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:53.992395 1174 raft_consensus.cc:2802] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:53.997335 1307 raft_consensus.cc:513] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:53.998009 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ac92f8b04a83459998241d6fd82297e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:53.998802 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac92f8b04a83459998241d6fd82297e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.000165 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e5d1377cfc84651afd53a78735468d8" } modify_external_catalogs: true
I20250625 13:51:54.000331 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e5d1377cfc84651afd53a78735468d8" } modify_external_catalogs: true
I20250625 13:51:54.001673 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_239d2674c30b4a6c9fe11163dc7f0b09" } new_table_name: "default.Table_5a2f402bff584ef78d361f827bcd56f7" modify_external_catalogs: true
I20250625 13:51:54.004549 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1231c5f6367b4046beaa57696ea42ff4" } modify_external_catalogs: true
I20250625 13:51:54.004772 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1231c5f6367b4046beaa57696ea42ff4" } modify_external_catalogs: true
I20250625 13:51:54.006294 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d8586363293040cc9ef927c2179bc1ae" } new_table_name: "default.Table_6cfe68ee62114f35be36b7f12d451a24" modify_external_catalogs: true
I20250625 13:51:54.007781 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da4f9a46a5804da3aff1c9a87b2e4984" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:54.007807 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da4f9a46a5804da3aff1c9a87b2e4984" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:54.008013 776 raft_consensus.cc:3058] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:54.008361 849 leader_election.cc:343] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.010857 1171 ts_tablet_manager.cc:1428] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.035s user 0.007s sys 0.004s
I20250625 13:51:54.011175 1171 tablet_bootstrap.cc:492] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:54.011715 776 raft_consensus.cc:2466] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:54.012634 1166 ts_tablet_manager.cc:1428] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.188s user 0.007s sys 0.003s
I20250625 13:51:54.013386 1171 tablet_bootstrap.cc:654] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.014122 1166 tablet_bootstrap.cc:492] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.014292 1307 leader_election.cc:290] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:54.016304 1166 tablet_bootstrap.cc:654] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.020026 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_bcac7658498546e39e2b1f2d30910433 [id=c2e58d3856d7401ca7b97d9cce92bbce]). Expected version 5 got 0
W20250625 13:51:54.021308 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0723e91650174f5badd74f7756c5a54e consensus state has no leader
I20250625 13:51:54.021760 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 921163dc5f454c9a97e72b005f9dba19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.022198 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 15750915ff7746ffa66509184be55dc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e0f2fa5d5614ad394e6e12ec62814d4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.022706 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 with delete_type TABLET_DATA_DELETED (replaced by tablet f00e72e103f443e886f0df690cff4b6f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.023139 889 tablet_service.cc:1515] Processing DeleteTablet for tablet ce84f74470ad428abb2113efa170ca24 with delete_type TABLET_DATA_DELETED (replaced by tablet 01ba67b500b646d7aa55315f7ee23ed6) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.023456 889 tablet_service.cc:1515] Processing DeleteTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5ba76c130a4711af5b21e9eb8562e7) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.023800 889 tablet_service.cc:1515] Processing DeleteTablet for tablet bf54376db991424b8de0e8f252fcb43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.024317 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da4f9a46a5804da3aff1c9a87b2e4984" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:54.024618 915 raft_consensus.cc:2391] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c1aa07ab6bc4b0089e46951240a37c9 in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:51:54.025135 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40a6375e98a04f95aa2bbde6b9b0228f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:54.025343 915 raft_consensus.cc:2466] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:54.029842 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 872724cd95bf473cb40e5f8ed8b5bd55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.030215 1167 tablet_replica.cc:331] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:54.032266 851 leader_election.cc:304] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:54.032920 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e02e9aad565644299f4613c5b807162c" } modify_external_catalogs: true
W20250625 13:51:54.033071 851 consensus_peers.cc:487] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92a90b0dc85844a892ab247ae0d09a12. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:54.033144 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e02e9aad565644299f4613c5b807162c" } modify_external_catalogs: true
I20250625 13:51:54.035571 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7108932fa8a241819ca90674a64b285e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:54.036542 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_df8e6bfa3a4741fca89b09188c645be9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.036953 711 leader_election.cc:343] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.037194 711 leader_election.cc:304] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
W20250625 13:51:54.037274 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df8e6bfa3a4741fca89b09188c645be9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:54.037505 711 leader_election.cc:343] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.038575 1303 raft_consensus.cc:2802] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:54.038729 1303 raft_consensus.cc:491] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:54.038976 1303 raft_consensus.cc:3058] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.039469 1308 raft_consensus.cc:2802] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:54.039747 1308 raft_consensus.cc:695] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:54.040122 1308 consensus_queue.cc:237] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:54.043576 985 leader_election.cc:343] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 872724cd95bf473cb40e5f8ed8b5bd55
W20250625 13:51:54.044684 985 leader_election.cc:343] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 872724cd95bf473cb40e5f8ed8b5bd55
I20250625 13:51:54.045941 1303 raft_consensus.cc:513] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.046559 1303 leader_election.cc:290] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:54.047063 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a1d1eec1b64d49d9a7c378dc4772a5db" } modify_external_catalogs: true
I20250625 13:51:54.047271 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a1d1eec1b64d49d9a7c378dc4772a5db" } modify_external_catalogs: true
I20250625 13:51:54.047297 1297 raft_consensus.cc:513] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.047885 1297 leader_election.cc:290] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:54.048218 1165 raft_consensus.cc:2270] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.048938 1165 ts_tablet_manager.cc:1905] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.053038 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7108932fa8a241819ca90674a64b285e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:54.053341 915 raft_consensus.cc:2466] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:54.054988 1310 tablet.cc:1722] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:51:54.056263 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f (DEFAULT_TABLE table=default.Table_72b53bc133ea42568572d075e5493c2e [id=df6a0a5985da4e6e82eb4b99149e24ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.056841 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e58fbec94e4ff9b7ac9830b576fe7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.067827 1030 tablet_service.cc:1468] Processing CreateTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 (DEFAULT_TABLE table=default.Table_77082a1ec23a48adb1f251559dc90002 [id=d482f250757e4eaaa5a5a4daa115bd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.068095 1024 tablet_service.cc:1468] Processing CreateTablet for tablet c911d9ecbded417daf70e85c7cbcd52e (DEFAULT_TABLE table=default.Table_a90cc30efd7145fdbaa7ebdc5db04107 [id=dec0cb5442804b6398a22dbd8c9b0f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.068225 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c5bea3b6214fa8af2792c5e9c411d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.068454 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c911d9ecbded417daf70e85c7cbcd52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.069783 889 tablet_service.cc:1468] Processing CreateTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f (DEFAULT_TABLE table=default.Table_72b53bc133ea42568572d075e5493c2e [id=df6a0a5985da4e6e82eb4b99149e24ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.070758 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e58fbec94e4ff9b7ac9830b576fe7f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:54.072804 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67441795e22b445a90c88d76aa519ea5 consensus state has no leader
I20250625 13:51:54.074349 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0723e91650174f5badd74f7756c5a54e
I20250625 13:51:54.074875 589 catalog_manager.cc:6652] ReplaceTablet: tablet 0723e91650174f5badd74f7756c5a54e deleted and replaced by tablet 34ab663b7eba452c9f9775c0f568010b
I20250625 13:51:54.076133 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 5786bb7716ef44689ef9862fc3ce17e9 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:54.077015 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14a12a5453e647b9946ca19e0ffb14ee consensus state has no leader
I20250625 13:51:54.078573 891 tablet_service.cc:1468] Processing CreateTablet for tablet c911d9ecbded417daf70e85c7cbcd52e (DEFAULT_TABLE table=default.Table_a90cc30efd7145fdbaa7ebdc5db04107 [id=dec0cb5442804b6398a22dbd8c9b0f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.079034 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c911d9ecbded417daf70e85c7cbcd52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.079129 889 tablet_service.cc:1468] Processing CreateTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 (DEFAULT_TABLE table=default.Table_77082a1ec23a48adb1f251559dc90002 [id=d482f250757e4eaaa5a5a4daa115bd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.079465 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c5bea3b6214fa8af2792c5e9c411d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.080350 747 tablet_service.cc:1468] Processing CreateTablet for tablet c911d9ecbded417daf70e85c7cbcd52e (DEFAULT_TABLE table=default.Table_a90cc30efd7145fdbaa7ebdc5db04107 [id=dec0cb5442804b6398a22dbd8c9b0f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.080847 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c911d9ecbded417daf70e85c7cbcd52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.084115 754 tablet_service.cc:1468] Processing CreateTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 (DEFAULT_TABLE table=default.Table_77082a1ec23a48adb1f251559dc90002 [id=d482f250757e4eaaa5a5a4daa115bd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.084604 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c5bea3b6214fa8af2792c5e9c411d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.090134 758 tablet_service.cc:1468] Processing CreateTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f (DEFAULT_TABLE table=default.Table_72b53bc133ea42568572d075e5493c2e [id=df6a0a5985da4e6e82eb4b99149e24ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.090651 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e58fbec94e4ff9b7ac9830b576fe7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.090790 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 0723e91650174f5badd74f7756c5a54e with delete_type TABLET_DATA_DELETED (replaced by tablet 34ab663b7eba452c9f9775c0f568010b) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.092832 593 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_cc3624b43cb14dfca4a12946b7198092 [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 1 got 0
I20250625 13:51:54.097249 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0723e91650174f5badd74f7756c5a54e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:54.097486 1047 raft_consensus.cc:3058] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.101101 1047 raft_consensus.cc:2466] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:54.102344 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40a6375e98a04f95aa2bbde6b9b0228f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:54.102690 1050 raft_consensus.cc:3058] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.105676 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 40a6375e98a04f95aa2bbde6b9b0228f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.106019 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet a179c3698d424ec7b6d87bf65cb858cc with delete_type TABLET_DATA_DELETED (replaced by tablet 0197f37545604f59b582ccbaa6a1a853) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.106657 1050 raft_consensus.cc:2466] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:54.107394 711 leader_election.cc:304] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:54.107811 711 leader_election.cc:304] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:54.108354 1302 raft_consensus.cc:2802] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:54.108521 1302 raft_consensus.cc:695] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:54.109169 1302 consensus_queue.cc:237] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:54.113999 851 leader_election.cc:343] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.114193 851 leader_election.cc:304] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:54.115252 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0723e91650174f5badd74f7756c5a54e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:54.115576 916 raft_consensus.cc:3058] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.118607 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 0723e91650174f5badd74f7756c5a54e with delete_type TABLET_DATA_DELETED (replaced by tablet 34ab663b7eba452c9f9775c0f568010b) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.121770 1024 tablet_service.cc:1468] Processing CreateTablet for tablet cf1664afbab647b3baeed6391196bed9 (DEFAULT_TABLE table=default.Table_98ae9c7310a24e8d988b919a5ec496e7 [id=05b9dac188b34d0b8022defddccbff26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.122385 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1664afbab647b3baeed6391196bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.133514 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40a6375e98a04f95aa2bbde6b9b0228f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:54.133798 915 raft_consensus.cc:3058] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.136276 1027 tablet_service.cc:1468] Processing CreateTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 (DEFAULT_TABLE table=default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.136847 1306 raft_consensus.cc:2802] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:54.136982 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd4f0857f52449e1b26fde09c6c7c9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.139156 884 tablet_service.cc:1468] Processing CreateTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 (DEFAULT_TABLE table=default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.142045 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 7d361e73fe3d4ea2bb6735f814345ebb (DEFAULT_TABLE table=default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.142128 883 tablet_service.cc:1468] Processing CreateTablet for tablet cf1664afbab647b3baeed6391196bed9 (DEFAULT_TABLE table=default.Table_98ae9c7310a24e8d988b919a5ec496e7 [id=05b9dac188b34d0b8022defddccbff26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.142710 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1664afbab647b3baeed6391196bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.142750 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d361e73fe3d4ea2bb6735f814345ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.145763 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9481589ea47c480192adfa47174a5511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.146737 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9481589ea47c480192adfa47174a5511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.149026 1174 raft_consensus.cc:695] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:54.149101 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7ccc261a6367449583a5c30495bfe2a9" } modify_external_catalogs: true
I20250625 13:51:54.149255 1306 raft_consensus.cc:695] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:54.149309 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7ccc261a6367449583a5c30495bfe2a9" } modify_external_catalogs: true
I20250625 13:51:54.149546 1174 consensus_queue.cc:237] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.151139 1308 raft_consensus.cc:2747] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:54.151436 751 tablet_service.cc:1468] Processing CreateTablet for tablet cf1664afbab647b3baeed6391196bed9 (DEFAULT_TABLE table=default.Table_98ae9c7310a24e8d988b919a5ec496e7 [id=05b9dac188b34d0b8022defddccbff26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.152025 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf1664afbab647b3baeed6391196bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.152238 752 tablet_service.cc:1468] Processing CreateTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 (DEFAULT_TABLE table=default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.152666 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd4f0857f52449e1b26fde09c6c7c9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.151957 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 0723e91650174f5badd74f7756c5a54e with delete_type TABLET_DATA_DELETED (replaced by tablet 34ab663b7eba452c9f9775c0f568010b) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.155200 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2c1a98dc40684954a602567cf3476879
I20250625 13:51:54.156052 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4212481baebd45429d103db60e844de5" } new_table_name: "default.Table_c3f37af0b4094d84909408f68b8b1314" modify_external_catalogs: true
I20250625 13:51:54.157383 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 799f2edd960d45c1ace9195283a042e2
I20250625 13:51:54.157768 591 catalog_manager.cc:6652] ReplaceTablet: tablet 799f2edd960d45c1ace9195283a042e2 deleted and replaced by tablet c7c3f0ca6fa5482db7f57143911c838f
I20250625 13:51:54.157989 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 921163dc5f454c9a97e72b005f9dba19 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:54.158434 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 8149aea0ff7646d3b94c9159f56e48ef from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:54.161136 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4b549bc8cead4af4a72f41f51655602c
W20250625 13:51:54.161120 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 53341.2us
I20250625 13:51:54.161976 592 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f923a448e99b4b57b642580808512e0f
I20250625 13:51:54.164613 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b7641869a0bb463698244ff445d87a1f
I20250625 13:51:54.165117 1168 ts_tablet_manager.cc:1918] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.165036 1306 consensus_queue.cc:237] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.165378 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a07c15abd0224e7d80d182ea2c891993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:54.165500 1168 log.cc:1199] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/39dde16d7f144ef880b52abf9c979a53
W20250625 13:51:54.165968 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a07c15abd0224e7d80d182ea2c891993 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.167829 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c1a98dc40684954a602567cf3476879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.168481 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 799f2edd960d45c1ace9195283a042e2 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c3f0ca6fa5482db7f57143911c838f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.169122 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b549bc8cead4af4a72f41f51655602c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.171136 1166 tablet_bootstrap.cc:492] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.171365 1166 ts_tablet_manager.cc:1397] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.157s user 0.000s sys 0.004s
I20250625 13:51:54.171866 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet b7641869a0bb463698244ff445d87a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.172120 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 799f2edd960d45c1ace9195283a042e2 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c3f0ca6fa5482db7f57143911c838f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.172281 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b549bc8cead4af4a72f41f51655602c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.172538 750 tablet_service.cc:1515] Processing DeleteTablet for tablet b7641869a0bb463698244ff445d87a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.173545 1166 raft_consensus.cc:357] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.174556 1166 raft_consensus.cc:383] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.174659 1166 raft_consensus.cc:738] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.174944 1166 consensus_queue.cc:260] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.175590 1166 ts_tablet_manager.cc:1428] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 13:51:54.176010 1166 tablet_bootstrap.cc:492] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:51:54.177412 849 consensus_peers.cc:487] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92a90b0dc85844a892ab247ae0d09a12. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:54.177985 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c1a98dc40684954a602567cf3476879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.178334 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c1a98dc40684954a602567cf3476879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.179240 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_747773f08f754220a96f903acd824389"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.179867 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_747773f08f754220a96f903acd824389 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.181325 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 799f2edd960d45c1ace9195283a042e2 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c3f0ca6fa5482db7f57143911c838f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.181696 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b549bc8cead4af4a72f41f51655602c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.182051 895 tablet_service.cc:1515] Processing DeleteTablet for tablet b7641869a0bb463698244ff445d87a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.184470 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bbd45439068f4c9188192e1b98a84784"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.185031 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbd45439068f4c9188192e1b98a84784 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.186319 752 tablet_service.cc:1468] Processing CreateTablet for tablet 7d361e73fe3d4ea2bb6735f814345ebb (DEFAULT_TABLE table=default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.187018 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d361e73fe3d4ea2bb6735f814345ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.187155 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2b3cf346c6724405a0e96eef702fc43e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.187759 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b3cf346c6724405a0e96eef702fc43e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.189262 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8910e07e05ad4d71bfff7d5a0d6952f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.189754 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8910e07e05ad4d71bfff7d5a0d6952f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.191752 711 leader_election.cc:304] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:54.192402 711 leader_election.cc:304] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:54.193661 750 tablet_service.cc:1468] Processing CreateTablet for tablet 643928ca4fc94ce58431a0525f2abb99 (DEFAULT_TABLE table=default.Table_ad4e07371e3b42c5b05954f9537fd6b7 [id=c861ee66493e47ec97ad3aa9a6495499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.193796 1302 raft_consensus.cc:2747] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:54.194067 1302 raft_consensus.cc:2802] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:54.194164 1302 raft_consensus.cc:491] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:54.194293 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643928ca4fc94ce58431a0525f2abb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.198580 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 04479dfb34584f0891c604ef72bf93b5 (DEFAULT_TABLE table=default.Table_c1e172984ca643a9b7c88d09f4d5221a [id=d0ac91ff8e7f473b8b2f938243067eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.199069 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04479dfb34584f0891c604ef72bf93b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.206386 757 tablet_service.cc:1468] Processing CreateTablet for tablet bb1c7a85da844d34be805843bc60d350 (DEFAULT_TABLE table=default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.206915 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1c7a85da844d34be805843bc60d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.208343 895 tablet_service.cc:1468] Processing CreateTablet for tablet 7d361e73fe3d4ea2bb6735f814345ebb (DEFAULT_TABLE table=default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.208781 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d361e73fe3d4ea2bb6735f814345ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.210511 893 tablet_service.cc:1468] Processing CreateTablet for tablet 04479dfb34584f0891c604ef72bf93b5 (DEFAULT_TABLE table=default.Table_c1e172984ca643a9b7c88d09f4d5221a [id=d0ac91ff8e7f473b8b2f938243067eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.212673 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 643928ca4fc94ce58431a0525f2abb99 (DEFAULT_TABLE table=default.Table_ad4e07371e3b42c5b05954f9537fd6b7 [id=c861ee66493e47ec97ad3aa9a6495499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.214182 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643928ca4fc94ce58431a0525f2abb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.219328 751 tablet_service.cc:1468] Processing CreateTablet for tablet 04479dfb34584f0891c604ef72bf93b5 (DEFAULT_TABLE table=default.Table_c1e172984ca643a9b7c88d09f4d5221a [id=d0ac91ff8e7f473b8b2f938243067eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.219764 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04479dfb34584f0891c604ef72bf93b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.223461 1030 tablet_service.cc:1468] Processing CreateTablet for tablet bb1c7a85da844d34be805843bc60d350 (DEFAULT_TABLE table=default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.223939 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1c7a85da844d34be805843bc60d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.224014 1302 raft_consensus.cc:3058] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.224270 896 tablet_service.cc:1468] Processing CreateTablet for tablet 643928ca4fc94ce58431a0525f2abb99 (DEFAULT_TABLE table=default.Table_ad4e07371e3b42c5b05954f9537fd6b7 [id=c861ee66493e47ec97ad3aa9a6495499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.224686 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643928ca4fc94ce58431a0525f2abb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.226296 894 tablet_service.cc:1468] Processing CreateTablet for tablet bb1c7a85da844d34be805843bc60d350 (DEFAULT_TABLE table=default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.226600 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1c7a85da844d34be805843bc60d350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.226718 1302 raft_consensus.cc:513] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:54.227773 581 consensus_peers.cc:487] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 -> Peer 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015): Couldn't send request to peer 3fec9d8a79054a5991ae90d8a3101e32. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 13:51:54.228560 1170 tablet_bootstrap.cc:492] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:54.228732 1170 ts_tablet_manager.cc:1397] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.416s user 0.103s sys 0.187s
I20250625 13:51:54.231941 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5786bb7716ef44689ef9862fc3ce17e9
I20250625 13:51:54.232441 589 catalog_manager.cc:6652] ReplaceTablet: tablet 5786bb7716ef44689ef9862fc3ce17e9 deleted and replaced by tablet d8b30c229c3445b683a5015f069eca1c
I20250625 13:51:54.233081 1170 raft_consensus.cc:357] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.233322 1170 raft_consensus.cc:383] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.233394 1170 raft_consensus.cc:738] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.234074 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0f9db08d3ae44bae911d94598e6e124c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.234678 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f9db08d3ae44bae911d94598e6e124c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.234807 916 raft_consensus.cc:2466] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:54.235234 1170 consensus_queue.cc:260] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.235751 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd4f0857f52449e1b26fde09c6c7c9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.235853 1170 ts_tablet_manager.cc:1428] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250625 13:51:54.236228 1170 tablet_bootstrap.cc:492] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:54.237972 1170 tablet_bootstrap.cc:654] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.238996 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 with delete_type TABLET_DATA_DELETED (replaced by tablet d8b30c229c3445b683a5015f069eca1c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.244500 1171 tablet_bootstrap.cc:492] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:54.244702 1171 ts_tablet_manager.cc:1397] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.234s user 0.059s sys 0.095s
I20250625 13:51:54.245679 1171 raft_consensus.cc:357] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.245903 1171 raft_consensus.cc:383] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.246001 1171 raft_consensus.cc:738] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.246230 1171 consensus_queue.cc:260] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.246917 915 raft_consensus.cc:2466] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:54.247762 1170 tablet_bootstrap.cc:492] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:54.247921 1170 ts_tablet_manager.cc:1397] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20250625 13:51:54.249241 1170 raft_consensus.cc:357] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.249445 1170 raft_consensus.cc:383] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.249536 1170 raft_consensus.cc:738] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.249789 1170 consensus_queue.cc:260] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.246917 1171 ts_tablet_manager.cc:1428] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:54.256084 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7108932fa8a241819ca90674a64b285e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:54.257046 916 raft_consensus.cc:3058] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.257150 1171 tablet_bootstrap.cc:492] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:54.257447 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04479dfb34584f0891c604ef72bf93b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.265856 916 raft_consensus.cc:2466] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:54.266371 711 leader_election.cc:304] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:54.268078 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 with delete_type TABLET_DATA_DELETED (replaced by tablet d8b30c229c3445b683a5015f069eca1c) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.268570 1303 raft_consensus.cc:2802] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:54.269035 1303 raft_consensus.cc:695] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:54.269376 1303 consensus_queue.cc:237] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.271147 1170 ts_tablet_manager.cc:1428] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.023s user 0.007s sys 0.006s
I20250625 13:51:54.271469 1170 tablet_bootstrap.cc:492] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:54.273327 1170 tablet_bootstrap.cc:654] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.273607 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 with delete_type TABLET_DATA_DELETED (replaced by tablet d8b30c229c3445b683a5015f069eca1c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.277648 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e930f596e0f6454c8af779d2e0f7ff76" } new_table_name: "default.Table_0f521fe0405e4837b9b41acdc5b12c51" modify_external_catalogs: true
W20250625 13:51:54.284547 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9bb7ebc40a14ecaa0c692a8e1117db3 consensus state has no leader
I20250625 13:51:54.284957 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3bb8047afb704209a73d570f4d255cc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.285576 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bb8047afb704209a73d570f4d255cc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.287293 1167 raft_consensus.cc:2241] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:54.287828 1167 pending_rounds.cc:70] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:51:54.287932 1167 pending_rounds.cc:77] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520569960480768 op_type: NO_OP noop_request { }
I20250625 13:51:54.288064 1167 raft_consensus.cc:2887] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:54.288203 1167 raft_consensus.cc:2270] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.288833 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7a3fe0cc014e4cfd9e21d44e013bfc8b" } new_table_name: "default.Table_1329da083f314d9d8fa3ac2e851646bf" modify_external_catalogs: true
I20250625 13:51:54.290540 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0191a529552e4cb3b2ed7b47c274b71b" } new_table_name: "default.Table_eccb37374f1d4d81947bc4ae4ee545f8" modify_external_catalogs: true
I20250625 13:51:54.292102 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8149aea0ff7646d3b94c9159f56e48ef
I20250625 13:51:54.292500 1167 ts_tablet_manager.cc:1905] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.292544 591 catalog_manager.cc:6652] ReplaceTablet: tablet 8149aea0ff7646d3b94c9159f56e48ef deleted and replaced by tablet 88f72c555d9842f28ca4bf310da9c53c
I20250625 13:51:54.293951 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7fdbbd3847ab4eb6b3f9872d5b0d9c4c" } new_table_name: "default.Table_180b708af9884741956d0cb3898f9efd" modify_external_catalogs: true
I20250625 13:51:54.299499 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef with delete_type TABLET_DATA_DELETED (replaced by tablet 88f72c555d9842f28ca4bf310da9c53c) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.300909 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:33015: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111),
I20250625 13:51:54.301059 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 13:51:54.301595 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 93f449c80b604a6498f6fd1d1ba4770a
I20250625 13:51:54.301895 1167 ts_tablet_manager.cc:1918] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:54.302062 1167 log.cc:1199] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/872724cd95bf473cb40e5f8ed8b5bd55
I20250625 13:51:54.302281 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_759e39cb7c154d6281c46789f97f0691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:54.302589 1167 ts_tablet_manager.cc:1939] T 872724cd95bf473cb40e5f8ed8b5bd55 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:51:54.302814 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_759e39cb7c154d6281c46789f97f0691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.304000 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.304564 1167 tablet_replica.cc:331] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:54.304893 1167 raft_consensus.cc:2241] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.305159 1167 raft_consensus.cc:2270] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.306080 1167 ts_tablet_manager.cc:1905] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.307644 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.307646 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 872724cd95bf473cb40e5f8ed8b5bd55 (table default.Table_c94d10947ecf49fba6f7608086ef35bc [id=263e68a7d9344016afdb8f4dc5b27b4f]) successfully deleted
I20250625 13:51:54.315611 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef with delete_type TABLET_DATA_DELETED (replaced by tablet 88f72c555d9842f28ca4bf310da9c53c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.322384 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.322829 1168 ts_tablet_manager.cc:1939] T 39dde16d7f144ef880b52abf9c979a53 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:54.323119 1171 tablet_bootstrap.cc:654] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.323987 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 81) took 1351 ms. Trace:
I20250625 13:51:54.324219 1168 rpcz_store.cc:276] 0625 13:51:52.972627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.082797 (+110170us) service_pool.cc:225] Handling call
0625 13:51:53.775071 (+692274us) ts_tablet_manager.cc:1145] Deleting tablet 39dde16d7f144ef880b52abf9c979a53
0625 13:51:53.775080 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:54.027586 (+252506us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.164931 (+137345us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.323941 (+159010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":691971}
I20250625 13:51:54.325018 597 master_service.cc:948] ReplaceTablet: received request to replace tablet da4f9a46a5804da3aff1c9a87b2e4984 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:54.325131 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 39dde16d7f144ef880b52abf9c979a53 (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]) successfully deleted
I20250625 13:51:54.325544 1168 tablet_replica.cc:331] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:54.325798 1168 raft_consensus.cc:2241] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.326045 1168 raft_consensus.cc:2270] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.327104 1168 ts_tablet_manager.cc:1905] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.327903 1302 leader_election.cc:290] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:54.328411 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7108932fa8a241819ca90674a64b285e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:54.328738 1050 raft_consensus.cc:3058] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.333427 1166 tablet_bootstrap.cc:654] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:54.334578 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50786a4c25274ffb8f9910c5dafe3933 consensus state has no leader
W20250625 13:51:54.337229 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a21335a06ec04da29b3351554c75f03c consensus state has no leader
W20250625 13:51:54.337817 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6771d8e180fe48569a0e016db0cc9b51 has no consensus state
W20250625 13:51:54.339956 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8b30c229c3445b683a5015f069eca1c has no consensus state
I20250625 13:51:54.343498 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet da4f9a46a5804da3aff1c9a87b2e4984 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5ba76c130a4711af5b21e9eb8562e7) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.345291 1165 ts_tablet_manager.cc:1918] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.345537 1165 log.cc:1199] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ca5dda8b27b34dafb20d8ed8afc58460
I20250625 13:51:54.346086 1165 ts_tablet_manager.cc:1939] T ca5dda8b27b34dafb20d8ed8afc58460 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:54.346731 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f033bfa4925b4d1c971789eb77efd92c (DEFAULT_TABLE table=default.Table_df8e6bfa3a4741fca89b09188c645be9 [id=e8ccbd9e2b83408b87689d6d6f26eb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.347240 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f033bfa4925b4d1c971789eb77efd92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.351933 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef with delete_type TABLET_DATA_DELETED (replaced by tablet 88f72c555d9842f28ca4bf310da9c53c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.352355 1170 tablet_bootstrap.cc:492] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:54.352550 1170 ts_tablet_manager.cc:1397] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.081s user 0.010s sys 0.037s
I20250625 13:51:54.332531 883 tablet_service.cc:1515] Processing DeleteTablet for tablet e75b5fb51373451a94cb5ca59cdcefac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.340283 1171 tablet_bootstrap.cc:492] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:54.352550 895 tablet_service.cc:1468] Processing CreateTablet for tablet f033bfa4925b4d1c971789eb77efd92c (DEFAULT_TABLE table=default.Table_df8e6bfa3a4741fca89b09188c645be9 [id=e8ccbd9e2b83408b87689d6d6f26eb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.353449 1171 ts_tablet_manager.cc:1397] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.002s
I20250625 13:51:54.353698 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f033bfa4925b4d1c971789eb77efd92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.354305 1171 raft_consensus.cc:357] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.355020 1171 raft_consensus.cc:383] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.355330 1171 raft_consensus.cc:738] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.355633 1171 consensus_queue.cc:260] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.356489 1165 tablet_replica.cc:331] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:54.356689 1165 raft_consensus.cc:2241] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.356956 1030 tablet_service.cc:1468] Processing CreateTablet for tablet b1acc41a93f6433588727811903921fe (DEFAULT_TABLE table=default.Table_ac92f8b04a83459998241d6fd82297e9 [id=380637ad0dc14b208e6b6840ded2d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.357388 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1acc41a93f6433588727811903921fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.357344 1170 raft_consensus.cc:357] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.356875 1165 raft_consensus.cc:2270] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.358004 1170 raft_consensus.cc:383] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.358237 1171 ts_tablet_manager.cc:1428] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:51:54.358902 1165 ts_tablet_manager.cc:1905] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.359645 883 tablet_service.cc:1468] Processing CreateTablet for tablet b1acc41a93f6433588727811903921fe (DEFAULT_TABLE table=default.Table_ac92f8b04a83459998241d6fd82297e9 [id=380637ad0dc14b208e6b6840ded2d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.360095 1171 tablet_bootstrap.cc:492] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:54.361902 1171 tablet_bootstrap.cc:654] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.363035 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1acc41a93f6433588727811903921fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.364491 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d675ca14ac884a35aab70e66cdcef846" } modify_external_catalogs: true
I20250625 13:51:54.364694 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d675ca14ac884a35aab70e66cdcef846" } modify_external_catalogs: true
I20250625 13:51:54.368094 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a90cc30efd7145fdbaa7ebdc5db04107" } modify_external_catalogs: true
I20250625 13:51:54.368135 1166 tablet_bootstrap.cc:492] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.368252 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a90cc30efd7145fdbaa7ebdc5db04107" } modify_external_catalogs: true
I20250625 13:51:54.368425 1166 ts_tablet_manager.cc:1397] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.193s user 0.003s sys 0.001s
I20250625 13:51:54.369222 1166 raft_consensus.cc:357] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.369419 1166 raft_consensus.cc:383] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.369514 1166 raft_consensus.cc:738] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.369776 1166 consensus_queue.cc:260] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.369968 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_df8e6bfa3a4741fca89b09188c645be9" } modify_external_catalogs: true
I20250625 13:51:54.370131 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_df8e6bfa3a4741fca89b09188c645be9" } modify_external_catalogs: true
I20250625 13:51:54.372018 1166 ts_tablet_manager.cc:1428] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:54.372315 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1329da083f314d9d8fa3ac2e851646bf" } modify_external_catalogs: true
I20250625 13:51:54.372491 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1329da083f314d9d8fa3ac2e851646bf" } modify_external_catalogs: true
I20250625 13:51:54.372993 1166 tablet_bootstrap.cc:492] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.374315 1166 tablet_bootstrap.cc:654] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.375422 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cc7e074a01254cbb9272fd4ea7dba124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:54.375628 1171 tablet_bootstrap.cc:492] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:54.375768 1171 ts_tablet_manager.cc:1397] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
W20250625 13:51:54.375924 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc7e074a01254cbb9272fd4ea7dba124 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.376529 1171 raft_consensus.cc:357] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.376713 1171 raft_consensus.cc:383] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.376772 1171 raft_consensus.cc:738] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.377002 1171 consensus_queue.cc:260] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.377489 1171 ts_tablet_manager.cc:1428] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:51:54.377821 1171 tablet_bootstrap.cc:492] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:54.379544 1171 tablet_bootstrap.cc:654] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.380164 1168 ts_tablet_manager.cc:1918] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:51:54.380292 1168 log.cc:1199] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/6d92711284e64f71b31666800c250eab
I20250625 13:51:54.383450 1166 tablet_bootstrap.cc:492] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.383674 1166 ts_tablet_manager.cc:1397] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20250625 13:51:54.384652 1166 raft_consensus.cc:357] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.384917 1166 raft_consensus.cc:383] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.385011 1166 raft_consensus.cc:738] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.385505 1168 ts_tablet_manager.cc:1939] T 6d92711284e64f71b31666800c250eab P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:54.386651 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 76) took 1437 ms. Trace:
I20250625 13:51:54.387955 1165 ts_tablet_manager.cc:1918] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.388149 1168 rpcz_store.cc:276] 0625 13:51:52.948883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.083146 (+134263us) service_pool.cc:225] Handling call
0625 13:51:54.325457 (+1242311us) ts_tablet_manager.cc:1145] Deleting tablet 6d92711284e64f71b31666800c250eab
0625 13:51:54.325470 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:54.364985 (+ 39515us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.380037 (+ 15052us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.386566 (+ 6529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241862}
I20250625 13:51:54.388769 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 6d92711284e64f71b31666800c250eab (table default.Table_2273827d860d46af8da96df2ffd22a04 [id=96be558f5b694f3483d2501659dc5097]) successfully deleted
I20250625 13:51:54.389691 1168 tablet_replica.cc:331] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:54.390004 1168 raft_consensus.cc:2241] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.390282 1165 log.cc:1199] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/da4f9a46a5804da3aff1c9a87b2e4984
W20250625 13:51:54.390328 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 6771d8e180fe48569a0e016db0cc9b51 (table default.Table_eccb37374f1d4d81947bc4ae4ee545f8 [id=65ec4be481574b20a64c98a75c6c4367], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6771d8e180fe48569a0e016db0cc9b51 consensus state has no leader
I20250625 13:51:54.391965 1165 ts_tablet_manager.cc:1939] T da4f9a46a5804da3aff1c9a87b2e4984 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:54.389256 1166 consensus_queue.cc:260] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.393004 1170 raft_consensus.cc:738] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.393687 1170 consensus_queue.cc:260] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.393496 1050 raft_consensus.cc:2466] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:54.395150 1166 ts_tablet_manager.cc:1428] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 13:51:54.395531 1166 tablet_bootstrap.cc:492] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.396330 1168 raft_consensus.cc:2270] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.397894 1170 ts_tablet_manager.cc:1428] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.042s user 0.001s sys 0.002s
I20250625 13:51:54.397924 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 991bb428def545fd979e9f1a80ce4d18 (DEFAULT_TABLE table=default.Table_a21873ae3bbd4a1aaf287e80fe14c12e [id=fd9a1350f16e4c54a613cf8f7026240a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.398515 883 tablet_service.cc:1468] Processing CreateTablet for tablet 055f88e118454e32baa578cae9aceb0c (DEFAULT_TABLE table=default.Table_59ed24a1a856411ba6d9c5504c4c9a7a [id=e50f8ace632e4ea284d17b85f474cef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.399025 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055f88e118454e32baa578cae9aceb0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:54.399669 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45727, user_credentials={real_user=slave}} blocked reactor thread for 66486.4us
I20250625 13:51:54.399888 1170 tablet_bootstrap.cc:492] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:54.400403 752 tablet_service.cc:1468] Processing CreateTablet for tablet f033bfa4925b4d1c971789eb77efd92c (DEFAULT_TABLE table=default.Table_df8e6bfa3a4741fca89b09188c645be9 [id=e8ccbd9e2b83408b87689d6d6f26eb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.400691 893 tablet_service.cc:1468] Processing CreateTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 (DEFAULT_TABLE table=default.Table_eccb37374f1d4d81947bc4ae4ee545f8 [id=65ec4be481574b20a64c98a75c6c4367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.400864 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f033bfa4925b4d1c971789eb77efd92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.401067 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6771d8e180fe48569a0e016db0cc9b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.403551 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 055f88e118454e32baa578cae9aceb0c (DEFAULT_TABLE table=default.Table_59ed24a1a856411ba6d9c5504c4c9a7a [id=e50f8ace632e4ea284d17b85f474cef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.408964 894 tablet_service.cc:1468] Processing CreateTablet for tablet 39ac79102eab46688e767faed6d6f114 (DEFAULT_TABLE table=default.Table_9798c064000341b5bc384b0dee1787d3 [id=62553d70f8ee470f944206ddc8cc2089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.408989 752 tablet_service.cc:1468] Processing CreateTablet for tablet b1acc41a93f6433588727811903921fe (DEFAULT_TABLE table=default.Table_ac92f8b04a83459998241d6fd82297e9 [id=380637ad0dc14b208e6b6840ded2d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.409431 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ac79102eab46688e767faed6d6f114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.409487 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1acc41a93f6433588727811903921fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.410522 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 991bb428def545fd979e9f1a80ce4d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.411670 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055f88e118454e32baa578cae9aceb0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.413116 896 tablet_service.cc:1468] Processing CreateTablet for tablet 46316eb725bc47e695af476c0b185504 (DEFAULT_TABLE table=default.Table_bc823b8c0cbd4aa8a4c7465e9e36410a [id=26d0bdba8be543d89d5337c86bbf6e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.413565 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46316eb725bc47e695af476c0b185504. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.415520 1308 raft_consensus.cc:491] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:54.415704 1308 raft_consensus.cc:513] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.415800 758 tablet_service.cc:1468] Processing CreateTablet for tablet 991bb428def545fd979e9f1a80ce4d18 (DEFAULT_TABLE table=default.Table_a21873ae3bbd4a1aaf287e80fe14c12e [id=fd9a1350f16e4c54a613cf8f7026240a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.416144 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 991bb428def545fd979e9f1a80ce4d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.416569 1308 leader_election.cc:290] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:54.421072 1170 tablet_bootstrap.cc:654] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.422937 758 tablet_service.cc:1468] Processing CreateTablet for tablet 055f88e118454e32baa578cae9aceb0c (DEFAULT_TABLE table=default.Table_59ed24a1a856411ba6d9c5504c4c9a7a [id=e50f8ace632e4ea284d17b85f474cef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.423311 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055f88e118454e32baa578cae9aceb0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.423985 1167 ts_tablet_manager.cc:1918] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.424121 1167 log.cc:1199] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/0723e91650174f5badd74f7756c5a54e
I20250625 13:51:54.424475 1167 ts_tablet_manager.cc:1939] T 0723e91650174f5badd74f7756c5a54e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:54.424790 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf54376db991424b8de0e8f252fcb43f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:54.424930 1167 tablet_replica.cc:331] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:54.425114 1167 raft_consensus.cc:2241] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.425302 1167 raft_consensus.cc:2270] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.425812 1167 ts_tablet_manager.cc:1905] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.426033 777 raft_consensus.cc:2466] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:54.433986 751 tablet_service.cc:1468] Processing CreateTablet for tablet 46316eb725bc47e695af476c0b185504 (DEFAULT_TABLE table=default.Table_bc823b8c0cbd4aa8a4c7465e9e36410a [id=26d0bdba8be543d89d5337c86bbf6e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.434466 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46316eb725bc47e695af476c0b185504. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.435639 754 tablet_service.cc:1468] Processing CreateTablet for tablet 39ac79102eab46688e767faed6d6f114 (DEFAULT_TABLE table=default.Table_9798c064000341b5bc384b0dee1787d3 [id=62553d70f8ee470f944206ddc8cc2089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.436018 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ac79102eab46688e767faed6d6f114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.439107 757 tablet_service.cc:1468] Processing CreateTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 (DEFAULT_TABLE table=default.Table_eccb37374f1d4d81947bc4ae4ee545f8 [id=65ec4be481574b20a64c98a75c6c4367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.439474 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6771d8e180fe48569a0e016db0cc9b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.443579 750 tablet_service.cc:1468] Processing CreateTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 (DEFAULT_TABLE table=default.Table_4755ed4ecc244e44a38ad91f159a119b [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.443939 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4a105e80c1460cbb21e1bdb9b356b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.451535 851 leader_election.cc:304] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:54.451920 1308 raft_consensus.cc:2802] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:54.452054 1308 raft_consensus.cc:491] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:54.452127 1308 raft_consensus.cc:3058] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.458473 892 tablet_service.cc:1468] Processing CreateTablet for tablet f00e72e103f443e886f0df690cff4b6f (DEFAULT_TABLE table=default.Table_b5472b871d6743cb80ba576b1cf02ee6 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.460044 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00e72e103f443e886f0df690cff4b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.461624 884 tablet_service.cc:1468] Processing CreateTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 (DEFAULT_TABLE table=default.Table_4755ed4ecc244e44a38ad91f159a119b [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.461668 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 36da1da539e24f55978d516873cb0b27 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:54.462025 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4a105e80c1460cbb21e1bdb9b356b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.458563 891 tablet_service.cc:1468] Processing CreateTablet for tablet 34ab663b7eba452c9f9775c0f568010b (DEFAULT_TABLE table=default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.464425 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab663b7eba452c9f9775c0f568010b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.467367 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 46316eb725bc47e695af476c0b185504 (DEFAULT_TABLE table=default.Table_bc823b8c0cbd4aa8a4c7465e9e36410a [id=26d0bdba8be543d89d5337c86bbf6e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.467541 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 (DEFAULT_TABLE table=default.Table_eccb37374f1d4d81947bc4ae4ee545f8 [id=65ec4be481574b20a64c98a75c6c4367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.467993 889 tablet_service.cc:1468] Processing CreateTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 (DEFAULT_TABLE table=default.Table_2c63842cb66d43c19e91b079c23ad1df [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.468072 1021 tablet_service.cc:1468] Processing CreateTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 (DEFAULT_TABLE table=default.Table_4755ed4ecc244e44a38ad91f159a119b [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.468351 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ba67b500b646d7aa55315f7ee23ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.468461 1019 tablet_service.cc:1468] Processing CreateTablet for tablet c7c3f0ca6fa5482db7f57143911c838f (DEFAULT_TABLE table=default.Table_12b254975e4a431ba57506a99aed12b4 [id=82115aa29a8f456b8d7c2f3d9caa17e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.468631 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 34ab663b7eba452c9f9775c0f568010b (DEFAULT_TABLE table=default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.468497 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 39ac79102eab46688e767faed6d6f114 (DEFAULT_TABLE table=default.Table_9798c064000341b5bc384b0dee1787d3 [id=62553d70f8ee470f944206ddc8cc2089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.469198 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 5d7704b38144429b9c95675be84a4730 (DEFAULT_TABLE table=default.Table_8c4176baf42b43658ec245c90d86cb98 [id=1786625fb4994b51a8b90c53e7e1714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.469341 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 27092991fcad475fabc77d1f8bab0582 (DEFAULT_TABLE table=default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba [id=540530d4619a4093b20d1bcba9ba99ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.469746 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf54376db991424b8de0e8f252fcb43f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:54.469861 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 9a31f10c8ca745b7ad635c622a265450 (DEFAULT_TABLE table=default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.458775 886 tablet_service.cc:1468] Processing CreateTablet for tablet 19017cc372af43b2b5074a8990375ede (DEFAULT_TABLE table=default.Table_d675ca14ac884a35aab70e66cdcef846 [id=fed9fb5422bc435ab14ed8f178cc95a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.470636 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19017cc372af43b2b5074a8990375ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.471216 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 (DEFAULT_TABLE table=default.Table_2c63842cb66d43c19e91b079c23ad1df [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.471740 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 19017cc372af43b2b5074a8990375ede (DEFAULT_TABLE table=default.Table_d675ca14ac884a35aab70e66cdcef846 [id=fed9fb5422bc435ab14ed8f178cc95a1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:54.471885 849 leader_election.cc:343] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.472213 1025 tablet_service.cc:1468] Processing CreateTablet for tablet f00e72e103f443e886f0df690cff4b6f (DEFAULT_TABLE table=default.Table_b5472b871d6743cb80ba576b1cf02ee6 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.473421 747 tablet_service.cc:1468] Processing CreateTablet for tablet 5d7704b38144429b9c95675be84a4730 (DEFAULT_TABLE table=default.Table_8c4176baf42b43658ec245c90d86cb98 [id=1786625fb4994b51a8b90c53e7e1714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.473568 752 tablet_service.cc:1468] Processing CreateTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 (DEFAULT_TABLE table=default.Table_2c63842cb66d43c19e91b079c23ad1df [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.473388 595 catalog_manager.cc:5582] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:51:54.473649 758 tablet_service.cc:1468] Processing CreateTablet for tablet 19017cc372af43b2b5074a8990375ede (DEFAULT_TABLE table=default.Table_d675ca14ac884a35aab70e66cdcef846 [id=fed9fb5422bc435ab14ed8f178cc95a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.473485 755 tablet_service.cc:1468] Processing CreateTablet for tablet f00e72e103f443e886f0df690cff4b6f (DEFAULT_TABLE table=default.Table_b5472b871d6743cb80ba576b1cf02ee6 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.473857 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7704b38144429b9c95675be84a4730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.474085 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00e72e103f443e886f0df690cff4b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.476365 749 tablet_service.cc:1468] Processing CreateTablet for tablet 9a31f10c8ca745b7ad635c622a265450 (DEFAULT_TABLE table=default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.476622 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19017cc372af43b2b5074a8990375ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.458966 887 tablet_service.cc:1468] Processing CreateTablet for tablet 5d7704b38144429b9c95675be84a4730 (DEFAULT_TABLE table=default.Table_8c4176baf42b43658ec245c90d86cb98 [id=1786625fb4994b51a8b90c53e7e1714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.477608 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7704b38144429b9c95675be84a4730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.459141 890 tablet_service.cc:1468] Processing CreateTablet for tablet c7c3f0ca6fa5482db7f57143911c838f (DEFAULT_TABLE table=default.Table_12b254975e4a431ba57506a99aed12b4 [id=82115aa29a8f456b8d7c2f3d9caa17e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.479602 753 tablet_service.cc:1468] Processing CreateTablet for tablet c7c3f0ca6fa5482db7f57143911c838f (DEFAULT_TABLE table=default.Table_12b254975e4a431ba57506a99aed12b4 [id=82115aa29a8f456b8d7c2f3d9caa17e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.479728 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c3f0ca6fa5482db7f57143911c838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.479954 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c3f0ca6fa5482db7f57143911c838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.459342 885 tablet_service.cc:1468] Processing CreateTablet for tablet 27092991fcad475fabc77d1f8bab0582 (DEFAULT_TABLE table=default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba [id=540530d4619a4093b20d1bcba9ba99ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.480381 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27092991fcad475fabc77d1f8bab0582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.459542 888 tablet_service.cc:1468] Processing CreateTablet for tablet 9a31f10c8ca745b7ad635c622a265450 (DEFAULT_TABLE table=default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.481005 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a31f10c8ca745b7ad635c622a265450. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.481151 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a7b72c2c464421eacfcf64008c3aea8 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:54.482084 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a31f10c8ca745b7ad635c622a265450. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.483606 756 tablet_service.cc:1468] Processing CreateTablet for tablet 27092991fcad475fabc77d1f8bab0582 (DEFAULT_TABLE table=default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba [id=540530d4619a4093b20d1bcba9ba99ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.483978 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27092991fcad475fabc77d1f8bab0582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.486716 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
I20250625 13:51:54.486757 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 39dde16d7f144ef880b52abf9c979a53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8149aea0ff7646d3b94c9159f56e48ef) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.487249 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ba67b500b646d7aa55315f7ee23ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.488747 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_683f20e141ca4122bee28d4e86aece20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.489876 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_683f20e141ca4122bee28d4e86aece20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.491629 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cc3624b43cb14dfca4a12946b7198092" } new_table_name: "default.Table_b2017c57e5c7416494c31f6dcefdc43d" modify_external_catalogs: true
I20250625 13:51:54.493630 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c54f67fabc114fc6bb555987a103adba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.494160 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c54f67fabc114fc6bb555987a103adba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.495296 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c8b06a7351fb437aa0440f6fa3898939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.495761 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8b06a7351fb437aa0440f6fa3898939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.501966 751 tablet_service.cc:1468] Processing CreateTablet for tablet 34ab663b7eba452c9f9775c0f568010b (DEFAULT_TABLE table=default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.503703 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet da4f9a46a5804da3aff1c9a87b2e4984 (table default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]) successfully deleted
I20250625 13:51:54.455263 1308 raft_consensus.cc:513] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.518306 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ba67b500b646d7aa55315f7ee23ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.518568 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c3f0ca6fa5482db7f57143911c838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.518786 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7704b38144429b9c95675be84a4730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.519016 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19017cc372af43b2b5074a8990375ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.520761 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00e72e103f443e886f0df690cff4b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.523437 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46316eb725bc47e695af476c0b185504. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.523674 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ac79102eab46688e767faed6d6f114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.524224 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6771d8e180fe48569a0e016db0cc9b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.524708 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27092991fcad475fabc77d1f8bab0582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.525413 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4a105e80c1460cbb21e1bdb9b356b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.527707 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 19017cc372af43b2b5074a8990375ede
I20250625 13:51:54.528457 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab663b7eba452c9f9775c0f568010b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.528524 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_71166441017840fa9a67752209689ee2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.529065 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71166441017840fa9a67752209689ee2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.529115 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a31f10c8ca745b7ad635c622a265450. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.530159 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 19017cc372af43b2b5074a8990375ede with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.530344 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c911d9ecbded417daf70e85c7cbcd52e
I20250625 13:51:54.530933 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 09a8e15d57f14702ba0b5b52a94fc7c4 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:54.531117 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c911d9ecbded417daf70e85c7cbcd52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.531488 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0f9db08d3ae44bae911d94598e6e124c" } modify_external_catalogs: true
I20250625 13:51:54.531646 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0f9db08d3ae44bae911d94598e6e124c" } modify_external_catalogs: true
I20250625 13:51:54.532755 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f033bfa4925b4d1c971789eb77efd92c
I20250625 13:51:54.533380 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1989ae5a6212423c8453c95e655e9a6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.533905 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1989ae5a6212423c8453c95e655e9a6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.535018 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a21335a06ec04da29b3351554c75f03c
I20250625 13:51:54.535645 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9dfb8c48d5144c808567a16a30cc199e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.536085 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dfb8c48d5144c808567a16a30cc199e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.540336 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 67441795e22b445a90c88d76aa519ea5 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:54.548763 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_13c3269d65ff41bc87efff42b2dce296"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.549366 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13c3269d65ff41bc87efff42b2dce296 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.549768 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 19017cc372af43b2b5074a8990375ede with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.551035 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0ae4786bc45842eea9376ff1f404d621" } new_table_name: "default.Table_41e093f1c53c404fb4743c97d30f7f31" modify_external_catalogs: true
I20250625 13:51:54.552567 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b5472b871d6743cb80ba576b1cf02ee6" } new_table_name: "default.Table_d7e04ac5a7b3491bb209bc40967a3059" modify_external_catalogs: true
I20250625 13:51:54.554388 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0d633ac15e6e4b14b8fda5e3845c6c6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.554877 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d633ac15e6e4b14b8fda5e3845c6c6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.555500 758 tablet_service.cc:1515] Processing DeleteTablet for tablet c911d9ecbded417daf70e85c7cbcd52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.555681 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_12b254975e4a431ba57506a99aed12b4" } new_table_name: "default.Table_ea8cb48993044c799983dc051e437621" modify_external_catalogs: true
I20250625 13:51:54.558676 1165 tablet_replica.cc:331] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:54.514124 1168 ts_tablet_manager.cc:1905] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.562433 895 tablet_service.cc:1468] Processing CreateTablet for tablet 991bb428def545fd979e9f1a80ce4d18 (DEFAULT_TABLE table=default.Table_a21873ae3bbd4a1aaf287e80fe14c12e [id=fd9a1350f16e4c54a613cf8f7026240a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.562932 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 991bb428def545fd979e9f1a80ce4d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.563233 1165 raft_consensus.cc:2241] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.563511 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_09cd14059fee4f4aaadc9c4e4bd5a249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.564124 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09cd14059fee4f4aaadc9c4e4bd5a249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.564278 1307 raft_consensus.cc:491] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:54.560685 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab663b7eba452c9f9775c0f568010b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.564010 757 tablet_service.cc:1515] Processing DeleteTablet for tablet f033bfa4925b4d1c971789eb77efd92c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.565376 1307 raft_consensus.cc:513] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.565956 1307 leader_election.cc:290] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:54.565945 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_eccb37374f1d4d81947bc4ae4ee545f8" } new_table_name: "default.Table_254fead6128d4cb6b544757eb751f82b" modify_external_catalogs: true
I20250625 13:51:54.567037 591 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7f9a1d971f04470d82a822dca0dc271d
I20250625 13:51:54.567225 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf54376db991424b8de0e8f252fcb43f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
W20250625 13:51:54.567922 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 36da1da539e24f55978d516873cb0b27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36da1da539e24f55978d516873cb0b27 already in progress: creating tablet
W20250625 13:51:54.568207 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 921163dc5f454c9a97e72b005f9dba19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 921163dc5f454c9a97e72b005f9dba19 already in progress: creating tablet
W20250625 13:51:54.568435 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 583ee2f2c66841939c93104efc36facb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 583ee2f2c66841939c93104efc36facb already in progress: creating tablet
I20250625 13:51:54.563201 1308 leader_election.cc:290] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:54.570094 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 19017cc372af43b2b5074a8990375ede with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:51:54.570173 1222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:51:54.570700 1222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:51:54.570843 1222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 13:51:54.571010 1171 tablet_bootstrap.cc:492] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:54.571134 1171 ts_tablet_manager.cc:1397] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.193s user 0.010s sys 0.024s
I20250625 13:51:54.572283 1171 raft_consensus.cc:357] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.572458 1171 raft_consensus.cc:383] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.572814 1171 raft_consensus.cc:738] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.572889 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b11b3157dd764f229831c3b8417460c6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:54.573052 1171 consensus_queue.cc:260] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.573822 1171 ts_tablet_manager.cc:1428] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:54.574517 890 tablet_service.cc:1515] Processing DeleteTablet for tablet f033bfa4925b4d1c971789eb77efd92c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.574750 896 tablet_service.cc:1515] Processing DeleteTablet for tablet c911d9ecbded417daf70e85c7cbcd52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.574985 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a21335a06ec04da29b3351554c75f03c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:51:54.576990 851 leader_election.cc:343] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.577122 754 tablet_service.cc:1515] Processing DeleteTablet for tablet a21335a06ec04da29b3351554c75f03c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.582336 777 raft_consensus.cc:3058] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.584967 777 raft_consensus.cc:2466] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:54.585201 747 tablet_service.cc:1515] Processing DeleteTablet for tablet bf54376db991424b8de0e8f252fcb43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.585538 851 leader_election.cc:304] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:51:54.586535 1222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:51:54.586630 1222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:51:54.586722 1222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:51:54.586782 1222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:51:54.588136 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 39dde16d7f144ef880b52abf9c979a53 with delete_type TABLET_DATA_DELETED (replaced by tablet 8149aea0ff7646d3b94c9159f56e48ef) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.588310 1165 raft_consensus.cc:2270] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.589251 1165 ts_tablet_manager.cc:1905] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.589627 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f033bfa4925b4d1c971789eb77efd92c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.590135 1171 tablet_bootstrap.cc:492] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:54.592267 1308 raft_consensus.cc:2802] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:54.592557 1308 raft_consensus.cc:695] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:54.591854 1171 tablet_bootstrap.cc:654] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.594322 1308 consensus_queue.cc:237] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.595060 1166 tablet_bootstrap.cc:654] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.603161 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b11b3157dd764f229831c3b8417460c6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:54.604537 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet a21335a06ec04da29b3351554c75f03c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.605667 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 76f074156559406c875e65796e6b5ee6 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:54.605783 849 leader_election.cc:343] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.606460 849 leader_election.cc:304] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:54.606354 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf54376db991424b8de0e8f252fcb43f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:54.606897 1307 raft_consensus.cc:2747] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:54.607511 849 leader_election.cc:343] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.608682 1165 ts_tablet_manager.cc:1918] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.608819 1165 log.cc:1199] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a179c3698d424ec7b6d87bf65cb858cc
I20250625 13:51:54.609311 1165 ts_tablet_manager.cc:1939] T a179c3698d424ec7b6d87bf65cb858cc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:54.616376 1165 tablet_replica.cc:331] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:54.616819 1165 raft_consensus.cc:2241] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.617264 1165 raft_consensus.cc:2270] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.617951 1165 ts_tablet_manager.cc:1905] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.625782 1165 ts_tablet_manager.cc:1918] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:51:54.626947 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f00e72e103f443e886f0df690cff4b6f consensus state has no leader
W20250625 13:51:54.627482 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41a7862601f04fd7a938f7e5d5a4c0f9 consensus state has no leader
I20250625 13:51:54.628624 1167 ts_tablet_manager.cc:1918] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.628780 1167 log.cc:1199] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/40a6375e98a04f95aa2bbde6b9b0228f
W20250625 13:51:54.628799 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c3f0ca6fa5482db7f57143911c838f consensus state has no leader
I20250625 13:51:54.629215 1167 ts_tablet_manager.cc:1939] T 40a6375e98a04f95aa2bbde6b9b0228f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:54.630362 1167 tablet_replica.cc:331] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:54.630492 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 40a6375e98a04f95aa2bbde6b9b0228f (table default.Table_70ae19ea7742453484d9525b11f16752 [id=c73f632380944ac9b8c2c181642922ad]) successfully deleted
W20250625 13:51:54.631486 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ce84f74470ad428abb2113efa170ca24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce84f74470ad428abb2113efa170ca24 already in progress: creating tablet
W20250625 13:51:54.631834 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 87327f74beb845768bbfd5ae6d2bcf88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87327f74beb845768bbfd5ae6d2bcf88 already in progress: creating tablet
I20250625 13:51:54.632457 1165 log.cc:1199] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/15750915ff7746ffa66509184be55dc4
I20250625 13:51:54.632635 1307 raft_consensus.cc:491] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:54.632844 1307 raft_consensus.cc:513] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.633390 1165 ts_tablet_manager.cc:1939] T 15750915ff7746ffa66509184be55dc4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:54.633680 1307 leader_election.cc:290] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:54.634367 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:54.634995 851 leader_election.cc:343] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.641634 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:54.641686 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 15750915ff7746ffa66509184be55dc4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]) successfully deleted
W20250625 13:51:54.642216 849 leader_election.cc:343] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.642366 849 leader_election.cc:304] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:54.643247 1166 tablet_bootstrap.cc:492] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.643435 1166 ts_tablet_manager.cc:1397] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.248s user 0.010s sys 0.007s
I20250625 13:51:54.643958 1171 tablet_bootstrap.cc:492] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:54.644093 1171 ts_tablet_manager.cc:1397] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.007s
I20250625 13:51:54.644440 1166 raft_consensus.cc:357] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.644670 1166 raft_consensus.cc:383] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.644773 1166 raft_consensus.cc:738] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.645038 1166 consensus_queue.cc:260] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.646298 1167 raft_consensus.cc:2241] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.646838 1171 raft_consensus.cc:357] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.647040 1171 raft_consensus.cc:383] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.647125 1171 raft_consensus.cc:738] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.647255 1308 raft_consensus.cc:2747] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:54.650126 1167 raft_consensus.cc:2270] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:54.650457 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6771d8e180fe48569a0e016db0cc9b51 consensus state has no leader
I20250625 13:51:54.651268 1168 ts_tablet_manager.cc:1918] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.652516 751 tablet_service.cc:1515] Processing DeleteTablet for tablet e75b5fb51373451a94cb5ca59cdcefac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.653283 1166 ts_tablet_manager.cc:1428] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 13:51:54.655745 1166 tablet_bootstrap.cc:492] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.657771 751 tablet_service.cc:1468] Processing CreateTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 (DEFAULT_TABLE table=default.Table_759e39cb7c154d6281c46789f97f0691 [id=5efdc94a9cf74a3daa545d4d93684eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.658147 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58aa3c21191e4bea9c5395d6e32b8717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.658624 1168 log.cc:1199] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/99090fee41d44946b6243b1b4bff3635
I20250625 13:51:54.659009 1168 ts_tablet_manager.cc:1939] T 99090fee41d44946b6243b1b4bff3635 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:54.659794 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 80) took 1710 ms. Trace:
I20250625 13:51:54.659905 1168 rpcz_store.cc:276] 0625 13:51:52.949337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.089410 (+140073us) service_pool.cc:225] Handling call
0625 13:51:54.389564 (+1300154us) ts_tablet_manager.cc:1145] Deleting tablet 99090fee41d44946b6243b1b4bff3635
0625 13:51:54.389577 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:54.637696 (+248119us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.651143 (+ 13447us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.659757 (+ 8614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299788}
I20250625 13:51:54.660223 1168 tablet_replica.cc:331] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:54.660413 1168 raft_consensus.cc:2241] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.660552 1168 raft_consensus.cc:2270] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.661120 1168 ts_tablet_manager.cc:1905] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.661922 884 tablet_service.cc:1468] Processing CreateTablet for tablet af05fba6fe2042aab5dd47b30e4dfcf3 (DEFAULT_TABLE table=default.Table_3bb8047afb704209a73d570f4d255cc6 [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.662266 891 tablet_service.cc:1468] Processing CreateTablet for tablet dbd2120b4b084691b6604305773e74f9 (DEFAULT_TABLE table=default.Table_8910e07e05ad4d71bfff7d5a0d6952f4 [id=2dd7357a611c4866913dfd01fedcbe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.662696 747 tablet_service.cc:1468] Processing CreateTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 (DEFAULT_TABLE table=default.Table_747773f08f754220a96f903acd824389 [id=d3d911b4c96949afa23f4f64e9b5bbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.663101 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11711546f0e14f9a9eabf003f49a69b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.663198 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbd2120b4b084691b6604305773e74f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.664089 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 54fa042947da4e1b89018640b46b271a (DEFAULT_TABLE table=default.Table_9481589ea47c480192adfa47174a5511 [id=22b45bb056454aba99f356fbc831479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.664836 886 tablet_service.cc:1468] Processing CreateTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 (DEFAULT_TABLE table=default.Table_747773f08f754220a96f903acd824389 [id=d3d911b4c96949afa23f4f64e9b5bbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.665166 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11711546f0e14f9a9eabf003f49a69b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.665773 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8b7062c34fd143bf86b743055a27db8c (DEFAULT_TABLE table=default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.666106 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7062c34fd143bf86b743055a27db8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.666278 883 tablet_service.cc:1468] Processing CreateTablet for tablet 54fa042947da4e1b89018640b46b271a (DEFAULT_TABLE table=default.Table_9481589ea47c480192adfa47174a5511 [id=22b45bb056454aba99f356fbc831479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.666584 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54fa042947da4e1b89018640b46b271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.666734 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b (DEFAULT_TABLE table=default.Table_2b3cf346c6724405a0e96eef702fc43e [id=4891101f0201404ab9d5c88c094ce3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.667269 758 tablet_service.cc:1468] Processing CreateTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b (DEFAULT_TABLE table=default.Table_2b3cf346c6724405a0e96eef702fc43e [id=4891101f0201404ab9d5c88c094ce3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.667655 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e29e7b55570483b80c6e7a1f227fa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.669502 750 tablet_service.cc:1468] Processing CreateTablet for tablet af05fba6fe2042aab5dd47b30e4dfcf3 (DEFAULT_TABLE table=default.Table_3bb8047afb704209a73d570f4d255cc6 [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.669871 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af05fba6fe2042aab5dd47b30e4dfcf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.671423 1308 raft_consensus.cc:491] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:54.671571 896 tablet_service.cc:1468] Processing CreateTablet for tablet 8b7062c34fd143bf86b743055a27db8c (DEFAULT_TABLE table=default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.671603 754 tablet_service.cc:1468] Processing CreateTablet for tablet dbd2120b4b084691b6604305773e74f9 (DEFAULT_TABLE table=default.Table_8910e07e05ad4d71bfff7d5a0d6952f4 [id=2dd7357a611c4866913dfd01fedcbe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.671597 1308 raft_consensus.cc:513] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.671939 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbd2120b4b084691b6604305773e74f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.671947 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7062c34fd143bf86b743055a27db8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.675608 749 tablet_service.cc:1468] Processing CreateTablet for tablet 54fa042947da4e1b89018640b46b271a (DEFAULT_TABLE table=default.Table_9481589ea47c480192adfa47174a5511 [id=22b45bb056454aba99f356fbc831479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.675676 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b (DEFAULT_TABLE table=default.Table_2b3cf346c6724405a0e96eef702fc43e [id=4891101f0201404ab9d5c88c094ce3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.676198 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54fa042947da4e1b89018640b46b271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.676204 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e29e7b55570483b80c6e7a1f227fa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.676658 1308 leader_election.cc:290] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:54.679646 756 tablet_service.cc:1468] Processing CreateTablet for tablet 88f72c555d9842f28ca4bf310da9c53c (DEFAULT_TABLE table=default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.679652 894 tablet_service.cc:1468] Processing CreateTablet for tablet 88f72c555d9842f28ca4bf310da9c53c (DEFAULT_TABLE table=default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.680011 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f72c555d9842f28ca4bf310da9c53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.680024 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f72c555d9842f28ca4bf310da9c53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.682866 1222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:51:54.683746 755 tablet_service.cc:1468] Processing CreateTablet for tablet d8b30c229c3445b683a5015f069eca1c (DEFAULT_TABLE table=default.Table_180b708af9884741956d0cb3898f9efd [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.683929 887 tablet_service.cc:1468] Processing CreateTablet for tablet d8b30c229c3445b683a5015f069eca1c (DEFAULT_TABLE table=default.Table_180b708af9884741956d0cb3898f9efd [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.684113 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b30c229c3445b683a5015f069eca1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.684870 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b30c229c3445b683a5015f069eca1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.684933 895 tablet_service.cc:1468] Processing CreateTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 (DEFAULT_TABLE table=default.Table_759e39cb7c154d6281c46789f97f0691 [id=5efdc94a9cf74a3daa545d4d93684eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.685312 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58aa3c21191e4bea9c5395d6e32b8717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.687112 1222 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:51:54.687613 892 tablet_service.cc:1468] Processing CreateTablet for tablet 637847cac1724bfeb629c4d4013396ca (DEFAULT_TABLE table=default.Table_a07c15abd0224e7d80d182ea2c891993 [id=47cd45886fc94cd7ba5cafa60838d601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.688019 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637847cac1724bfeb629c4d4013396ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.691196 1222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 13:51:54.651717 1171 consensus_queue.cc:260] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.669188 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 88f72c555d9842f28ca4bf310da9c53c (DEFAULT_TABLE table=default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.694334 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 (DEFAULT_TABLE table=default.Table_759e39cb7c154d6281c46789f97f0691 [id=5efdc94a9cf74a3daa545d4d93684eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.694756 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 (DEFAULT_TABLE table=default.Table_747773f08f754220a96f903acd824389 [id=d3d911b4c96949afa23f4f64e9b5bbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.695058 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 76f074156559406c875e65796e6b5ee6
I20250625 13:51:54.695098 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54fa042947da4e1b89018640b46b271a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.695506 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15750915ff7746ffa66509184be55dc4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:54.696959 1024 tablet_service.cc:1468] Processing CreateTablet for tablet af05fba6fe2042aab5dd47b30e4dfcf3 (DEFAULT_TABLE table=default.Table_3bb8047afb704209a73d570f4d255cc6 [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.697890 752 tablet_service.cc:1468] Processing CreateTablet for tablet 637847cac1724bfeb629c4d4013396ca (DEFAULT_TABLE table=default.Table_a07c15abd0224e7d80d182ea2c891993 [id=47cd45886fc94cd7ba5cafa60838d601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.698280 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637847cac1724bfeb629c4d4013396ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.698319 588 catalog_manager.cc:6652] ReplaceTablet: tablet 76f074156559406c875e65796e6b5ee6 deleted and replaced by tablet 6d95a376304e464ab75e2e1e00e7a924
I20250625 13:51:54.700881 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 76f074156559406c875e65796e6b5ee6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d95a376304e464ab75e2e1e00e7a924) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.701542 1171 ts_tablet_manager.cc:1428] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
W20250625 13:51:54.702242 851 leader_election.cc:343] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 15750915ff7746ffa66509184be55dc4
I20250625 13:51:54.703732 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 76f074156559406c875e65796e6b5ee6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d95a376304e464ab75e2e1e00e7a924) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.703893 1170 tablet_bootstrap.cc:492] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:54.703923 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 67441795e22b445a90c88d76aa519ea5
I20250625 13:51:54.704047 1021 tablet_service.cc:1468] Processing CreateTablet for tablet dbd2120b4b084691b6604305773e74f9 (DEFAULT_TABLE table=default.Table_8910e07e05ad4d71bfff7d5a0d6952f4 [id=2dd7357a611c4866913dfd01fedcbe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.704412 596 catalog_manager.cc:6652] ReplaceTablet: tablet 67441795e22b445a90c88d76aa519ea5 deleted and replaced by tablet abd6f9121a9d4b39adde1b880a875e23
I20250625 13:51:54.704855 1171 tablet_bootstrap.cc:492] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:54.706115 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 67441795e22b445a90c88d76aa519ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet abd6f9121a9d4b39adde1b880a875e23) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.707438 1171 tablet_bootstrap.cc:654] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.710536 1167 ts_tablet_manager.cc:1905] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.711289 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d8b30c229c3445b683a5015f069eca1c (DEFAULT_TABLE table=default.Table_180b708af9884741956d0cb3898f9efd [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.711928 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 67441795e22b445a90c88d76aa519ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet abd6f9121a9d4b39adde1b880a875e23) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.712390 1170 ts_tablet_manager.cc:1397] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.306s user 0.011s sys 0.014s
I20250625 13:51:54.712894 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e29e7b55570483b80c6e7a1f227fa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.717283 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 637847cac1724bfeb629c4d4013396ca (DEFAULT_TABLE table=default.Table_a07c15abd0224e7d80d182ea2c891993 [id=47cd45886fc94cd7ba5cafa60838d601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.718307 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 76f074156559406c875e65796e6b5ee6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d95a376304e464ab75e2e1e00e7a924) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.717643 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 8b7062c34fd143bf86b743055a27db8c (DEFAULT_TABLE table=default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.718824 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11711546f0e14f9a9eabf003f49a69b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.722057 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f72c555d9842f28ca4bf310da9c53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.723178 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 67441795e22b445a90c88d76aa519ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet abd6f9121a9d4b39adde1b880a875e23) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.723734 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af05fba6fe2042aab5dd47b30e4dfcf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.726639 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58aa3c21191e4bea9c5395d6e32b8717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.726603 883 tablet_service.cc:1468] Processing CreateTablet for tablet 29be8c0350584de382985e66556bdcca (DEFAULT_TABLE table=default.Table_13c3269d65ff41bc87efff42b2dce296 [id=1c58b83050f1494a91b0845e8dfe04bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.728339 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 29be8c0350584de382985e66556bdcca (DEFAULT_TABLE table=default.Table_13c3269d65ff41bc87efff42b2dce296 [id=1c58b83050f1494a91b0845e8dfe04bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.728715 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b7062c34fd143bf86b743055a27db8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.730016 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b (DEFAULT_TABLE table=default.Table_71166441017840fa9a67752209689ee2 [id=fc9bcf7791ec4c069f0bcae18943cac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.730351 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637847cac1724bfeb629c4d4013396ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.731139 888 tablet_service.cc:1468] Processing CreateTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b (DEFAULT_TABLE table=default.Table_71166441017840fa9a67752209689ee2 [id=fc9bcf7791ec4c069f0bcae18943cac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.731226 1022 tablet_service.cc:1468] Processing CreateTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc (DEFAULT_TABLE table=default.Table_1989ae5a6212423c8453c95e655e9a6a [id=8a37f6c60b414741a615d1fa7f9a4f3c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:54.733651 849 leader_election.cc:343] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:54.733816 849 leader_election.cc:304] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:54.733829 1170 raft_consensus.cc:357] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.734021 1170 raft_consensus.cc:383] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.734107 1170 raft_consensus.cc:738] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.734316 1170 consensus_queue.cc:260] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.734835 1170 ts_tablet_manager.cc:1428] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20250625 13:51:54.735162 1170 tablet_bootstrap.cc:492] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:54.735239 1308 raft_consensus.cc:2747] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:54.735589 885 tablet_service.cc:1468] Processing CreateTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc (DEFAULT_TABLE table=default.Table_1989ae5a6212423c8453c95e655e9a6a [id=8a37f6c60b414741a615d1fa7f9a4f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.736325 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a22ade8cefac4140b08882c311ea5181 (DEFAULT_TABLE table=default.Table_9dfb8c48d5144c808567a16a30cc199e [id=f55ecdcac2234415a085be00cacad571]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.736557 889 tablet_service.cc:1468] Processing CreateTablet for tablet a22ade8cefac4140b08882c311ea5181 (DEFAULT_TABLE table=default.Table_9dfb8c48d5144c808567a16a30cc199e [id=f55ecdcac2234415a085be00cacad571]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.738073 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af05fba6fe2042aab5dd47b30e4dfcf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.739926 596 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]). Expected version 1 got 0
I20250625 13:51:54.741568 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29be8c0350584de382985e66556bdcca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.743911 753 tablet_service.cc:1468] Processing CreateTablet for tablet a22ade8cefac4140b08882c311ea5181 (DEFAULT_TABLE table=default.Table_9dfb8c48d5144c808567a16a30cc199e [id=f55ecdcac2234415a085be00cacad571]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.744078 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_72b53bc133ea42568572d075e5493c2e" } modify_external_catalogs: true
I20250625 13:51:54.744288 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_72b53bc133ea42568572d075e5493c2e" } modify_external_catalogs: true
I20250625 13:51:54.744344 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22ade8cefac4140b08882c311ea5181. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.745736 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_77082a1ec23a48adb1f251559dc90002" } modify_external_catalogs: true
I20250625 13:51:54.745910 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_77082a1ec23a48adb1f251559dc90002" } modify_external_catalogs: true
I20250625 13:51:54.747165 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c15b19b72fc640e38bed77044f2acecd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.747725 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c15b19b72fc640e38bed77044f2acecd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.748682 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbd2120b4b084691b6604305773e74f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.749931 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]). Expected version 1 got 0
I20250625 13:51:54.750392 752 tablet_service.cc:1468] Processing CreateTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc (DEFAULT_TABLE table=default.Table_1989ae5a6212423c8453c95e655e9a6a [id=8a37f6c60b414741a615d1fa7f9a4f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.750851 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c666ff2d6b9b4ed4a18e370826c143fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.751868 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_09cd14059fee4f4aaadc9c4e4bd5a249" } modify_external_catalogs: true
I20250625 13:51:54.752048 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_09cd14059fee4f4aaadc9c4e4bd5a249" } modify_external_catalogs: true
I20250625 13:51:54.753918 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9481589ea47c480192adfa47174a5511" } new_table_name: "default.Table_753dc763991a4d2c900b8f1dc5e6138a" modify_external_catalogs: true
I20250625 13:51:54.755232 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet ca5dda8b27b34dafb20d8ed8afc58460 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b549bc8cead4af4a72f41f51655602c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.755507 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7a4c3ced3be545189e17b6f63f336f65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.756069 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a4c3ced3be545189e17b6f63f336f65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.757279 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22ade8cefac4140b08882c311ea5181. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.758127 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6264bd28054f4c52a39e6f6a6a4199bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:54.758589 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6cfe68ee62114f35be36b7f12d451a24" } modify_external_catalogs: true
I20250625 13:51:54.758766 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6cfe68ee62114f35be36b7f12d451a24" } modify_external_catalogs: true
W20250625 13:51:54.758661 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6264bd28054f4c52a39e6f6a6a4199bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.760190 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ae448e34759d455882fac1dfe9789add"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.760687 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae448e34759d455882fac1dfe9789add in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.763612 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f065b4e3218b4e6e8a2ac3a70eede13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.768734 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c666ff2d6b9b4ed4a18e370826c143fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.770044 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c666ff2d6b9b4ed4a18e370826c143fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.770318 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22ade8cefac4140b08882c311ea5181. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.773751 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f065b4e3218b4e6e8a2ac3a70eede13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.779596 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cd4b8059082844b99439152f1be88d22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.780345 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd4b8059082844b99439152f1be88d22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.785630 757 tablet_service.cc:1468] Processing CreateTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b (DEFAULT_TABLE table=default.Table_71166441017840fa9a67752209689ee2 [id=fc9bcf7791ec4c069f0bcae18943cac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.786118 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f065b4e3218b4e6e8a2ac3a70eede13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.799790 750 tablet_service.cc:1468] Processing CreateTablet for tablet 29be8c0350584de382985e66556bdcca (DEFAULT_TABLE table=default.Table_13c3269d65ff41bc87efff42b2dce296 [id=1c58b83050f1494a91b0845e8dfe04bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.800251 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29be8c0350584de382985e66556bdcca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.781853 1168 ts_tablet_manager.cc:1918] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.802491 1168 log.cc:1199] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/b11b3157dd764f229831c3b8417460c6
I20250625 13:51:54.803522 1168 ts_tablet_manager.cc:1939] T b11b3157dd764f229831c3b8417460c6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:54.804566 1165 tablet_replica.cc:331] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:54.804868 1165 raft_consensus.cc:2241] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:54.805158 1165 pending_rounds.cc:70] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:54.805248 1165 pending_rounds.cc:77] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520570022645760 op_type: NO_OP noop_request { }
I20250625 13:51:54.805401 1165 raft_consensus.cc:2887] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:54.805514 1165 raft_consensus.cc:2270] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:54.806198 851 consensus_peers.cc:487] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da4f9a46a5804da3aff1c9a87b2e4984. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:54.806399 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 90) took 1693 ms. Trace:
I20250625 13:51:54.807655 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8b30c229c3445b683a5015f069eca1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.808804 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c0a3ea18bab944749d45da9b3139f8bc (DEFAULT_TABLE table=default.Table_c8b06a7351fb437aa0440f6fa3898939 [id=eac16bb1f501401c9f6803136dc04116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.808815 1168 rpcz_store.cc:276] 0625 13:51:53.112771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.148778 (+ 36007us) service_pool.cc:225] Handling call
0625 13:51:54.660167 (+1511389us) ts_tablet_manager.cc:1145] Deleting tablet b11b3157dd764f229831c3b8417460c6
0625 13:51:54.660177 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:54.777904 (+117727us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.781742 (+ 3838us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.806347 (+ 24605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511120}
I20250625 13:51:54.809223 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet b11b3157dd764f229831c3b8417460c6 (table default.Table_8fc2c990732d4e658580c2bc2cf67f4d [id=75b34bf006e040d18266d0c90986afeb]) successfully deleted
I20250625 13:51:54.809310 752 tablet_service.cc:1468] Processing CreateTablet for tablet c0a3ea18bab944749d45da9b3139f8bc (DEFAULT_TABLE table=default.Table_c8b06a7351fb437aa0440f6fa3898939 [id=eac16bb1f501401c9f6803136dc04116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.808604 892 tablet_service.cc:1468] Processing CreateTablet for tablet c0a3ea18bab944749d45da9b3139f8bc (DEFAULT_TABLE table=default.Table_c8b06a7351fb437aa0440f6fa3898939 [id=eac16bb1f501401c9f6803136dc04116]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.809689 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a3ea18bab944749d45da9b3139f8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.809782 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a3ea18bab944749d45da9b3139f8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.809916 1168 tablet_replica.cc:331] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:54.810715 1168 raft_consensus.cc:2241] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:54.811524 1168 raft_consensus.cc:2270] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:54.812331 751 tablet_service.cc:1468] Processing CreateTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 (DEFAULT_TABLE table=default.Table_cc7e074a01254cbb9272fd4ea7dba124 [id=d85d6e75f7e9420096456eac1ab144cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.812458 747 tablet_service.cc:1468] Processing CreateTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 (DEFAULT_TABLE table=default.Table_683f20e141ca4122bee28d4e86aece20 [id=40f54158a1c84f5988762e7a44e3eab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.812824 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb9a98e402b4da8a9c2bd8317cef532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.812911 1168 ts_tablet_manager.cc:1905] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.815424 1015 tablet_service.cc:1468] Processing CreateTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 (DEFAULT_TABLE table=default.Table_cc7e074a01254cbb9272fd4ea7dba124 [id=d85d6e75f7e9420096456eac1ab144cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.816556 1342 tablet.cc:1722] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:51:54.816926 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 2cd5303e674246ada7d36574b535b333 (DEFAULT_TABLE table=default.Table_09cd14059fee4f4aaadc9c4e4bd5a249 [id=a79073f0368145dcab2528663f56e9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.818121 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bbd509e3dfb45d98314c6afe37677f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.821646 1166 tablet_bootstrap.cc:700] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent opening tablet: real 0.165s user 0.001s sys 0.000s
I20250625 13:51:54.822106 1166 tablet_bootstrap.cc:654] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.823194 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29be8c0350584de382985e66556bdcca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.824579 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb9a98e402b4da8a9c2bd8317cef532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.826140 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd5303e674246ada7d36574b535b333. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.827006 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d0c5bea3b6214fa8af2792c5e9c411d9
I20250625 13:51:54.827275 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a3ea18bab944749d45da9b3139f8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.827569 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 15e58fbec94e4ff9b7ac9830b576fe7f
I20250625 13:51:54.828246 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1a06c7c260544cce91ab86dad2d62346" } modify_external_catalogs: true
I20250625 13:51:54.828413 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1a06c7c260544cce91ab86dad2d62346" } modify_external_catalogs: true
I20250625 13:51:54.830348 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_374c2aaf889f4542893d3a66ff791a90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:54.830508 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:54.830871 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_374c2aaf889f4542893d3a66ff791a90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.831288 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 (DEFAULT_TABLE table=default.Table_683f20e141ca4122bee28d4e86aece20 [id=40f54158a1c84f5988762e7a44e3eab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.831522 895 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.832325 1341 tablet.cc:1722] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:51:54.834015 895 tablet_service.cc:1468] Processing CreateTablet for tablet 2cd5303e674246ada7d36574b535b333 (DEFAULT_TABLE table=default.Table_09cd14059fee4f4aaadc9c4e4bd5a249 [id=a79073f0368145dcab2528663f56e9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.834685 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd5303e674246ada7d36574b535b333. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.835054 890 tablet_service.cc:1468] Processing CreateTablet for tablet 653267b468ef4958b98af0ed6a7bf4d3 (DEFAULT_TABLE table=default.Table_0d633ac15e6e4b14b8fda5e3845c6c6d [id=6bfad46432b7473f88d77bfcfa2a4e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.835959 896 tablet_service.cc:1468] Processing CreateTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 (DEFAULT_TABLE table=default.Table_c54f67fabc114fc6bb555987a103adba [id=aa481ec0dc6e42b391641f4c3d05ca42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.836452 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653267b468ef4958b98af0ed6a7bf4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.838124 755 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.840341 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd39216b01054ff38ec9175d7a2f87f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.846670 756 tablet_service.cc:1468] Processing CreateTablet for tablet 2cd5303e674246ada7d36574b535b333 (DEFAULT_TABLE table=default.Table_09cd14059fee4f4aaadc9c4e4bd5a249 [id=a79073f0368145dcab2528663f56e9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.847121 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cd5303e674246ada7d36574b535b333. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.847299 1166 tablet_bootstrap.cc:492] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.847524 1166 ts_tablet_manager.cc:1397] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.192s user 0.003s sys 0.001s
I20250625 13:51:54.849016 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.849292 1166 raft_consensus.cc:357] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.849581 1166 raft_consensus.cc:383] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.849754 1166 raft_consensus.cc:738] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.849699 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ec42501f607c41ceb9d35f3dd2a38f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:54.850159 1166 consensus_queue.cc:260] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
W20250625 13:51:54.850253 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec42501f607c41ceb9d35f3dd2a38f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.854817 758 tablet_service.cc:1468] Processing CreateTablet for tablet 653267b468ef4958b98af0ed6a7bf4d3 (DEFAULT_TABLE table=default.Table_0d633ac15e6e4b14b8fda5e3845c6c6d [id=6bfad46432b7473f88d77bfcfa2a4e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.855403 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653267b468ef4958b98af0ed6a7bf4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.855913 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.855960 1047 raft_consensus.cc:3058] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:54.855036 1166 ts_tablet_manager.cc:1428] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 13:51:54.861029 1166 tablet_bootstrap.cc:492] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.864176 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2cd5303e674246ada7d36574b535b333
I20250625 13:51:54.865712 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cd5303e674246ada7d36574b535b333 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:54.866266 1166 tablet_bootstrap.cc:654] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.867185 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cd5303e674246ada7d36574b535b333 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.856374 749 tablet_service.cc:1468] Processing CreateTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 (DEFAULT_TABLE table=default.Table_c54f67fabc114fc6bb555987a103adba [id=aa481ec0dc6e42b391641f4c3d05ca42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.869220 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd39216b01054ff38ec9175d7a2f87f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.873446 1166 tablet_bootstrap.cc:492] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.873494 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1989ae5a6212423c8453c95e655e9a6a" } new_table_name: "default.Table_badc8f5a2d5040af96b9e7c9758c6417" modify_external_catalogs: true
I20250625 13:51:54.873737 1166 ts_tablet_manager.cc:1397] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20250625 13:51:54.874531 1166 raft_consensus.cc:357] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.876350 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 32221a413957450fa59ffcfbc91c51b9 with delete_type TABLET_DATA_DELETED (replaced by tablet bb1c7a85da844d34be805843bc60d350) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.876928 1166 raft_consensus.cc:383] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.877214 1166 raft_consensus.cc:738] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.877592 1166 consensus_queue.cc:260] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.880935 1166 ts_tablet_manager.cc:1428] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250625 13:51:54.881662 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54fa042947da4e1b89018640b46b271a consensus state has no leader
I20250625 13:51:54.882119 1166 tablet_bootstrap.cc:492] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.883764 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bbd509e3dfb45d98314c6afe37677f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.886380 1166 tablet_bootstrap.cc:654] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.888252 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_747773f08f754220a96f903acd824389" } new_table_name: "default.Table_1956bf8628234a2fa1b29b76594f624b" modify_external_catalogs: true
I20250625 13:51:54.890946 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 653267b468ef4958b98af0ed6a7bf4d3 (DEFAULT_TABLE table=default.Table_0d633ac15e6e4b14b8fda5e3845c6c6d [id=6bfad46432b7473f88d77bfcfa2a4e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.895419 1166 tablet_bootstrap.cc:492] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.895637 1166 ts_tablet_manager.cc:1397] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20250625 13:51:54.896684 1166 raft_consensus.cc:357] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.896958 1166 raft_consensus.cc:383] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.897112 1166 raft_consensus.cc:738] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.897456 1166 consensus_queue.cc:260] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.899915 1174 raft_consensus.cc:491] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:54.900061 1174 raft_consensus.cc:513] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.900905 1174 leader_election.cc:290] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:54.905237 985 leader_election.cc:343] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: da4f9a46a5804da3aff1c9a87b2e4984
I20250625 13:51:54.905956 1166 ts_tablet_manager.cc:1428] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 13:51:54.908051 1166 tablet_bootstrap.cc:492] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.910633 1166 tablet_bootstrap.cc:654] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.914412 758 tablet_service.cc:1515] Processing DeleteTablet for tablet ce84f74470ad428abb2113efa170ca24 with delete_type TABLET_DATA_DELETED (replaced by tablet 01ba67b500b646d7aa55315f7ee23ed6) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.914376 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d2296c79c2f949fca8a57149e361225e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.915210 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2296c79c2f949fca8a57149e361225e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.921726 1166 tablet_bootstrap.cc:492] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.921954 1166 ts_tablet_manager.cc:1397] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20250625 13:51:54.922690 1166 raft_consensus.cc:357] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.922948 1166 raft_consensus.cc:383] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.923048 1166 raft_consensus.cc:738] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.923358 1166 consensus_queue.cc:260] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.926774 1166 ts_tablet_manager.cc:1428] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 13:51:54.928790 1166 tablet_bootstrap.cc:492] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.931071 1166 tablet_bootstrap.cc:654] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.940409 1166 tablet_bootstrap.cc:492] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.940675 1166 ts_tablet_manager.cc:1397] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250625 13:51:54.940745 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]). Expected version 1 got 0
I20250625 13:51:54.941591 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3bb8047afb704209a73d570f4d255cc6" } new_table_name: "default.Table_cc6c89513ac9408bbfb6387793e10d33" modify_external_catalogs: true
I20250625 13:51:54.941648 1166 raft_consensus.cc:357] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.942026 1166 raft_consensus.cc:383] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.942164 1166 raft_consensus.cc:738] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.941922 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653267b468ef4958b98af0ed6a7bf4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.942530 1166 consensus_queue.cc:260] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.945191 1166 ts_tablet_manager.cc:1428] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:51:54.946350 1166 tablet_bootstrap.cc:492] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.948629 1166 tablet_bootstrap.cc:654] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.961342 1166 tablet_bootstrap.cc:492] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:54.961613 1166 ts_tablet_manager.cc:1397] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20250625 13:51:54.962831 1166 raft_consensus.cc:357] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.963143 1166 raft_consensus.cc:383] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:54.963266 1166 raft_consensus.cc:738] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:54.963641 1166 consensus_queue.cc:260] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.966114 1166 ts_tablet_manager.cc:1428] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:54.967321 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 14a12a5453e647b9946ca19e0ffb14ee
I20250625 13:51:54.970523 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.942260 1027 tablet_service.cc:1468] Processing CreateTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 (DEFAULT_TABLE table=default.Table_c54f67fabc114fc6bb555987a103adba [id=aa481ec0dc6e42b391641f4c3d05ca42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.973634 747 tablet_service.cc:1515] Processing DeleteTablet for tablet d0cf3d979369419882a2ba2083933d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.975485 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da4f9a46a5804da3aff1c9a87b2e4984" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:54.976258 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 99090fee41d44946b6243b1b4bff3635 with delete_type TABLET_DATA_DELETED (replaced by tablet d524089e6ab044ec961cfc9f587e8f86) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.973160 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c15b19b72fc640e38bed77044f2acecd" } new_table_name: "default.Table_29924a11586f4d2bb3d0f8574a7849f8" modify_external_catalogs: true
I20250625 13:51:54.976775 1165 ts_tablet_manager.cc:1905] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:54.981607 1166 tablet_bootstrap.cc:492] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:54.981916 1308 raft_consensus.cc:491] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:54.982087 1308 raft_consensus.cc:513] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:54.982406 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cd5303e674246ada7d36574b535b333 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.982846 1308 leader_election.cc:290] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:54.986433 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_62ab081749614851902de10929497600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:54.986985 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62ab081749614851902de10929497600 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:54.989090 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce84f74470ad428abb2113efa170ca24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:54.989949 983 leader_election.cc:304] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:54.990166 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:54.991456 1174 raft_consensus.cc:2747] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:51:54.993116 1170 tablet_bootstrap.cc:654] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.993278 1166 tablet_bootstrap.cc:654] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:54.988678 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce84f74470ad428abb2113efa170ca24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:54.989751 1302 raft_consensus.cc:491] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:54.993958 1302 raft_consensus.cc:513] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:54.994771 1302 leader_election.cc:290] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:54.989924 758 tablet_service.cc:1515] Processing DeleteTablet for tablet b11b3157dd764f229831c3b8417460c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 85c72f9ae92a4537979fb479eb4467c4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:54.995771 778 raft_consensus.cc:2466] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:54.995692 1307 consensus_queue.cc:1035] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.094s
I20250625 13:51:54.997454 1168 ts_tablet_manager.cc:1918] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:54.997406 889 tablet_service.cc:1468] Processing CreateTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 (DEFAULT_TABLE table=default.Table_cc7e074a01254cbb9272fd4ea7dba124 [id=d85d6e75f7e9420096456eac1ab144cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:54.997646 1168 log.cc:1199] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/40a6375e98a04f95aa2bbde6b9b0228f
I20250625 13:51:54.997918 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb9a98e402b4da8a9c2bd8317cef532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:54.998072 1168 ts_tablet_manager.cc:1939] T 40a6375e98a04f95aa2bbde6b9b0228f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:54.998905 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 104) took 1561 ms. Trace:
I20250625 13:51:54.999037 1168 rpcz_store.cc:276] 0625 13:51:53.437775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.443817 (+ 6042us) service_pool.cc:225] Handling call
0625 13:51:54.809829 (+1366012us) ts_tablet_manager.cc:1145] Deleting tablet 40a6375e98a04f95aa2bbde6b9b0228f
0625 13:51:54.809839 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:54.823704 (+ 13865us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.981731 (+158027us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:54.998855 (+ 17124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365768}
I20250625 13:51:54.999294 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39dde16d7f144ef880b52abf9c979a53" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:54.999523 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 103) took 1577 ms. Trace:
I20250625 13:51:54.999655 1168 rpcz_store.cc:276] 0625 13:51:53.421680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.444891 (+ 23211us) service_pool.cc:225] Handling call
0625 13:51:54.999355 (+1554464us) ts_tablet_manager.cc:1145] Deleting tablet 32221a413957450fa59ffcfbc91c51b9
0625 13:51:54.999364 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:54.999470 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1554271}
I20250625 13:51:55.000059 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 105) took 1548 ms. Trace:
I20250625 13:51:55.000201 1168 rpcz_store.cc:276] 0625 13:51:53.451873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.453317 (+ 1444us) service_pool.cc:225] Handling call
0625 13:51:54.999896 (+1546579us) ts_tablet_manager.cc:1145] Deleting tablet 600d711975a74f66b9144758ad0dcfb0
0625 13:51:54.999905 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.000015 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527864}
I20250625 13:51:55.000605 1168 tablet_replica.cc:331] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.000813 1168 raft_consensus.cc:2241] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.001125 1168 pending_rounds.cc:70] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:51:55.001212 1168 pending_rounds.cc:77] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520565854539776 op_type: NO_OP noop_request { }
I20250625 13:51:55.001317 1168 raft_consensus.cc:2887] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:55.001428 1168 raft_consensus.cc:2270] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.002048 1168 ts_tablet_manager.cc:1905] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.002574 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4256942aa0684fa3be579ab3ceb83fcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:55.003091 1167 ts_tablet_manager.cc:1918] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:51:55.003211 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4256942aa0684fa3be579ab3ceb83fcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:55.003597 849 leader_election.cc:343] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.006737 1166 tablet_bootstrap.cc:492] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.006974 1166 ts_tablet_manager.cc:1397] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.004s
W20250625 13:51:55.011021 849 consensus_peers.cc:487] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:55.012063 1166 raft_consensus.cc:357] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.012852 1166 raft_consensus.cc:383] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.013000 1166 raft_consensus.cc:738] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.013283 1166 consensus_queue.cc:260] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.014689 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 40a6375e98a04f95aa2bbde6b9b0228f (table default.Table_70ae19ea7742453484d9525b11f16752 [id=c73f632380944ac9b8c2c181642922ad]) successfully deleted
W20250625 13:51:55.015013 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 32221a413957450fa59ffcfbc91c51b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32221a413957450fa59ffcfbc91c51b9 already in progress: creating tablet
W20250625 13:51:55.015239 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 600d711975a74f66b9144758ad0dcfb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 600d711975a74f66b9144758ad0dcfb0 already in progress: creating tablet
I20250625 13:51:55.015877 1166 ts_tablet_manager.cc:1428] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 13:51:55.016362 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 389d81329b8d430a9098b0630612fdcb
I20250625 13:51:55.017308 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.017488 851 leader_election.cc:304] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.018227 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 389d81329b8d430a9098b0630612fdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.014130 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 (DEFAULT_TABLE table=default.Table_683f20e141ca4122bee28d4e86aece20 [id=40f54158a1c84f5988762e7a44e3eab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.019420 1307 raft_consensus.cc:2802] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.019975 1307 raft_consensus.cc:491] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.020099 1307 raft_consensus.cc:3058] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.024637 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 6771d8e180fe48569a0e016db0cc9b51 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.026499 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bbd509e3dfb45d98314c6afe37677f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.028684 1166 tablet_bootstrap.cc:492] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.030452 1302 consensus_queue.cc:1035] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:51:55.031473 711 leader_election.cc:343] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 39dde16d7f144ef880b52abf9c979a53
I20250625 13:51:55.034659 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.034971 1048 raft_consensus.cc:2466] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:55.035255 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd39216b01054ff38ec9175d7a2f87f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.035610 711 leader_election.cc:304] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters: affa1c3035b940e88ca143be7d35840d
I20250625 13:51:55.036109 1302 raft_consensus.cc:2802] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.036239 1302 raft_consensus.cc:491] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.036324 1302 raft_consensus.cc:3058] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.037530 1167 log.cc:1199] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a179c3698d424ec7b6d87bf65cb858cc
I20250625 13:51:55.037937 1167 ts_tablet_manager.cc:1939] T a179c3698d424ec7b6d87bf65cb858cc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.039129 1302 raft_consensus.cc:513] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.039625 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a179c3698d424ec7b6d87bf65cb858cc (table default.Table_05f514ab26624c719af36996404ef419 [id=2c4408e7c06545faadb7f7b49a846782]) successfully deleted
W20250625 13:51:55.039907 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2c1a98dc40684954a602567cf3476879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c1a98dc40684954a602567cf3476879 already in progress: creating tablet
W20250625 13:51:55.040153 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 799f2edd960d45c1ace9195283a042e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 799f2edd960d45c1ace9195283a042e2 already in progress: creating tablet
W20250625 13:51:55.040355 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4b549bc8cead4af4a72f41f51655602c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b549bc8cead4af4a72f41f51655602c already in progress: creating tablet
W20250625 13:51:55.040578 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b7641869a0bb463698244ff445d87a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7641869a0bb463698244ff445d87a1f already in progress: creating tablet
I20250625 13:51:55.040795 1167 tablet_replica.cc:331] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:51:55.040798 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5786bb7716ef44689ef9862fc3ce17e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5786bb7716ef44689ef9862fc3ce17e9 already in progress: creating tablet
W20250625 13:51:55.041025 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 93f449c80b604a6498f6fd1d1ba4770a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93f449c80b604a6498f6fd1d1ba4770a already in progress: creating tablet
W20250625 13:51:55.041204 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8149aea0ff7646d3b94c9159f56e48ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8149aea0ff7646d3b94c9159f56e48ef already in progress: creating tablet
I20250625 13:51:55.041260 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 389d81329b8d430a9098b0630612fdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.042506 1302 leader_election.cc:290] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.043259 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39dde16d7f144ef880b52abf9c979a53" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:55.043509 1048 raft_consensus.cc:3058] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.044154 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 389d81329b8d430a9098b0630612fdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.045162 1166 tablet_bootstrap.cc:654] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.045706 595 master_service.cc:948] ReplaceTablet: received request to replace tablet b7641869a0bb463698244ff445d87a1f from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.046232 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 8149aea0ff7646d3b94c9159f56e48ef from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.046593 1048 raft_consensus.cc:2466] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:55.046679 595 master_service.cc:948] ReplaceTablet: received request to replace tablet f00e72e103f443e886f0df690cff4b6f from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.047734 1307 raft_consensus.cc:513] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.048053 711 leader_election.cc:304] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:55.048545 1307 leader_election.cc:290] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.048663 1302 raft_consensus.cc:2802] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.048933 1302 raft_consensus.cc:695] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:55.049218 1302 consensus_queue.cc:237] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.051337 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 921163dc5f454c9a97e72b005f9dba19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:55.051446 711 leader_election.cc:343] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 39dde16d7f144ef880b52abf9c979a53
I20250625 13:51:55.052423 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce84f74470ad428abb2113efa170ca24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:51:55.054106 1359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:55.055444 1347 tablet.cc:1722] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 13:51:55.056258 1356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:51:55.057325 849 leader_election.cc:343] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.059509 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce84f74470ad428abb2113efa170ca24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:55.059791 778 raft_consensus.cc:3058] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.061291 1171 tablet_bootstrap.cc:492] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.061492 1171 ts_tablet_manager.cc:1397] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.357s user 0.018s sys 0.061s
I20250625 13:51:55.062521 1171 raft_consensus.cc:357] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.062732 1171 raft_consensus.cc:383] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.063107 778 raft_consensus.cc:2466] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:55.064473 1171 raft_consensus.cc:738] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.064785 1171 consensus_queue.cc:260] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.067502 851 leader_election.cc:304] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.068535 1171 ts_tablet_manager.cc:1428] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:51:55.068861 1171 tablet_bootstrap.cc:492] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.073599 1307 raft_consensus.cc:2802] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:51:55.073810 851 consensus_peers.cc:487] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a179c3698d424ec7b6d87bf65cb858cc. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:55.074568 1307 raft_consensus.cc:695] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:55.077064 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c1e172984ca643a9b7c88d09f4d5221a" } modify_external_catalogs: true
I20250625 13:51:55.077311 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c1e172984ca643a9b7c88d09f4d5221a" } modify_external_catalogs: true
I20250625 13:51:55.081103 1307 consensus_queue.cc:237] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.081934 1171 tablet_bootstrap.cc:654] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.090591 1168 ts_tablet_manager.cc:1918] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.090603 1165 ts_tablet_manager.cc:1918] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.090755 1168 log.cc:1199] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/92a90b0dc85844a892ab247ae0d09a12
I20250625 13:51:55.090785 1165 log.cc:1199] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/40a6375e98a04f95aa2bbde6b9b0228f
I20250625 13:51:55.091298 1165 ts_tablet_manager.cc:1939] T 40a6375e98a04f95aa2bbde6b9b0228f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:55.091385 1168 ts_tablet_manager.cc:1939] T 92a90b0dc85844a892ab247ae0d09a12 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.092209 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 106) took 1622 ms. Trace:
I20250625 13:51:55.092367 1168 rpcz_store.cc:276] 0625 13:51:53.470177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.471877 (+ 1700us) service_pool.cc:225] Handling call
0625 13:51:55.000454 (+1528577us) ts_tablet_manager.cc:1145] Deleting tablet 92a90b0dc85844a892ab247ae0d09a12
0625 13:51:55.000464 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.062412 (+ 61948us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.090300 (+ 27888us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.092166 (+ 1866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517450}
I20250625 13:51:55.092943 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 107) took 1605 ms. Trace:
I20250625 13:51:55.093032 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 92a90b0dc85844a892ab247ae0d09a12 (table default.Table_85c33778738445ae9981ea2f1902e0ec [id=caf5ae6d8d9c482491f4b6d21c421f26]) successfully deleted
I20250625 13:51:55.093137 1168 rpcz_store.cc:276] 0625 13:51:53.487365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.487781 (+ 416us) service_pool.cc:225] Handling call
0625 13:51:55.092800 (+1605019us) ts_tablet_manager.cc:1145] Deleting tablet d524089e6ab044ec961cfc9f587e8f86
0625 13:51:55.092810 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.092912 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1604772}
W20250625 13:51:55.094524 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d524089e6ab044ec961cfc9f587e8f86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d524089e6ab044ec961cfc9f587e8f86 already in progress: creating tablet
I20250625 13:51:55.094655 1167 raft_consensus.cc:2241] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.094377 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 108) took 1584 ms. Trace:
I20250625 13:51:55.095422 1168 rpcz_store.cc:276] 0625 13:51:53.510206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.510257 (+ 51us) service_pool.cc:225] Handling call
0625 13:51:55.094233 (+1583976us) ts_tablet_manager.cc:1145] Deleting tablet cbaa018753d8455199f3869323fb5582
0625 13:51:55.094241 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.094347 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583784}
I20250625 13:51:55.095736 1168 tablet_replica.cc:331] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:51:55.095914 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet cbaa018753d8455199f3869323fb5582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbaa018753d8455199f3869323fb5582 already in progress: creating tablet
I20250625 13:51:55.095959 1168 raft_consensus.cc:2241] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.096127 1168 raft_consensus.cc:2270] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.096685 1168 ts_tablet_manager.cc:1905] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.097254 1167 pending_rounds.cc:70] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:51:55.097409 1167 pending_rounds.cc:77] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520569512329216 op_type: NO_OP noop_request { }
I20250625 13:51:55.097623 1167 raft_consensus.cc:2887] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:55.097819 1167 raft_consensus.cc:2270] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:55.099455 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11711546f0e14f9a9eabf003f49a69b7 consensus state has no leader
W20250625 13:51:55.100881 849 consensus_peers.cc:487] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:55.100932 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8fc2c990732d4e658580c2bc2cf67f4d" } modify_external_catalogs: true
I20250625 13:51:55.101101 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8fc2c990732d4e658580c2bc2cf67f4d" } modify_external_catalogs: true
W20250625 13:51:55.101703 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c666ff2d6b9b4ed4a18e370826c143fc consensus state has no leader
I20250625 13:51:55.102142 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f41a4c203a604e7786bb8716dde80fd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.102845 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f41a4c203a604e7786bb8716dde80fd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.107435 1170 tablet_bootstrap.cc:492] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.107642 1170 ts_tablet_manager.cc:1397] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.373s user 0.002s sys 0.011s
I20250625 13:51:55.108968 1170 raft_consensus.cc:357] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.109222 1170 raft_consensus.cc:383] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.109313 1170 raft_consensus.cc:738] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.111183 1170 consensus_queue.cc:260] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.111958 1170 ts_tablet_manager.cc:1428] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:55.113351 1167 ts_tablet_manager.cc:1905] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.121344 1171 tablet_bootstrap.cc:492] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.121549 1171 ts_tablet_manager.cc:1397] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.012s
I20250625 13:51:55.122327 1171 raft_consensus.cc:357] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.122530 1171 raft_consensus.cc:383] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.122627 1171 raft_consensus.cc:738] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.122889 1171 consensus_queue.cc:260] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.124440 1171 ts_tablet_manager.cc:1428] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:51:55.124795 1171 tablet_bootstrap.cc:492] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.126680 1171 tablet_bootstrap.cc:654] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.127189 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_374c2aaf889f4542893d3a66ff791a90" } modify_external_catalogs: true
I20250625 13:51:55.127638 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_374c2aaf889f4542893d3a66ff791a90" } modify_external_catalogs: true
W20250625 13:51:55.131095 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 54381us
I20250625 13:51:55.134812 1170 tablet_bootstrap.cc:492] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.136977 1170 tablet_bootstrap.cc:654] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.096858 1303 raft_consensus.cc:491] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.137569 1367 raft_consensus.cc:491] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.137537 1303 raft_consensus.cc:513] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.137835 1367 raft_consensus.cc:513] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.138228 1303 leader_election.cc:290] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.139008 1367 leader_election.cc:290] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.139179 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "412a0c5f05ab489ab9f8fc3f44fa7bdc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.139542 778 raft_consensus.cc:2466] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:55.133776 1166 tablet_bootstrap.cc:492] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.139923 1166 ts_tablet_manager.cc:1397] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.017s
I20250625 13:51:55.140133 851 leader_election.cc:304] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:55.140385 1367 raft_consensus.cc:2802] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.140487 1367 raft_consensus.cc:491] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.140663 1367 raft_consensus.cc:3058] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.140843 1166 raft_consensus.cc:357] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.142143 1166 raft_consensus.cc:383] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.142235 1166 raft_consensus.cc:738] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.142494 1166 consensus_queue.cc:260] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.143003 1166 ts_tablet_manager.cc:1428] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 13:51:55.141889 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 134) took 1227 ms. Trace:
I20250625 13:51:55.144141 1367 raft_consensus.cc:513] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.145975 1367 leader_election.cc:290] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.146241 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "412a0c5f05ab489ab9f8fc3f44fa7bdc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:55.146498 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "412a0c5f05ab489ab9f8fc3f44fa7bdc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:55.146754 778 raft_consensus.cc:3058] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:55.146757 849 leader_election.cc:343] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.146955 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "412a0c5f05ab489ab9f8fc3f44fa7bdc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:55.147466 849 leader_election.cc:343] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.148147 1168 ts_tablet_manager.cc:1918] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.143824 1165 rpcz_store.cc:276] 0625 13:51:53.914581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.915183 (+ 602us) service_pool.cc:225] Handling call
0625 13:51:54.804481 (+889298us) ts_tablet_manager.cc:1145] Deleting tablet 40a6375e98a04f95aa2bbde6b9b0228f
0625 13:51:54.804490 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:54.981072 (+176582us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.090448 (+109376us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.141820 (+ 51372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":840551}
I20250625 13:51:55.149467 1166 tablet_bootstrap.cc:492] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.152046 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 40a6375e98a04f95aa2bbde6b9b0228f (table default.Table_70ae19ea7742453484d9525b11f16752 [id=c73f632380944ac9b8c2c181642922ad]) successfully deleted
I20250625 13:51:55.154428 1167 ts_tablet_manager.cc:1918] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.154503 1168 log.cc:1199] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a17f862a8a584047a8f0220f2eced869
I20250625 13:51:55.154780 1167 log.cc:1199] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/da4f9a46a5804da3aff1c9a87b2e4984
I20250625 13:51:55.155350 1167 ts_tablet_manager.cc:1939] T da4f9a46a5804da3aff1c9a87b2e4984 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.156149 1168 ts_tablet_manager.cc:1939] T a17f862a8a584047a8f0220f2eced869 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:51:55.156419 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af05fba6fe2042aab5dd47b30e4dfcf3 consensus state has no leader
I20250625 13:51:55.157493 1171 tablet_bootstrap.cc:492] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.157668 1171 ts_tablet_manager.cc:1397] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.007s
I20250625 13:51:55.158052 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 111) took 1528 ms. Trace:
I20250625 13:51:55.158648 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet da4f9a46a5804da3aff1c9a87b2e4984 (table default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]) successfully deleted
I20250625 13:51:55.157490 1170 tablet_bootstrap.cc:492] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.159633 1170 ts_tablet_manager.cc:1397] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250625 13:51:55.160925 1167 tablet_replica.cc:331] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:55.161116 1167 raft_consensus.cc:2241] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.161549 1168 rpcz_store.cc:276] 0625 13:51:53.629700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.629747 (+ 47us) service_pool.cc:225] Handling call
0625 13:51:55.095667 (+1465920us) ts_tablet_manager.cc:1145] Deleting tablet a17f862a8a584047a8f0220f2eced869
0625 13:51:55.095676 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.119913 (+ 24237us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.148020 (+ 28107us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.157993 (+ 9973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465692}
I20250625 13:51:55.162714 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet a17f862a8a584047a8f0220f2eced869 (table default.Table_72ad58769a804f88b2ae43dd40dbf4d3 [id=433b0870be4a44c5bf0fbc2bbf3b5e4a]) successfully deleted
I20250625 13:51:55.163244 1168 tablet_replica.cc:331] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.165931 1166 tablet_bootstrap.cc:654] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.161301 1167 raft_consensus.cc:2270] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.158512 1171 raft_consensus.cc:357] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.166560 1171 raft_consensus.cc:383] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.166702 1171 raft_consensus.cc:738] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.166736 1168 raft_consensus.cc:2241] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.167006 1171 consensus_queue.cc:260] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.168126 1168 raft_consensus.cc:2270] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.168180 1171 ts_tablet_manager.cc:1428] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20250625 13:51:55.168898 1171 tablet_bootstrap.cc:492] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.169951 1167 ts_tablet_manager.cc:1905] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.170683 1168 ts_tablet_manager.cc:1905] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.167472 1170 raft_consensus.cc:357] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.171948 1170 raft_consensus.cc:383] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.172783 1171 tablet_bootstrap.cc:654] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.173354 1170 raft_consensus.cc:738] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.173727 1170 consensus_queue.cc:260] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.179142 1170 ts_tablet_manager.cc:1428] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250625 13:51:55.184893 1170 tablet_bootstrap.cc:492] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.185554 1171 tablet_bootstrap.cc:492] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.185751 1171 ts_tablet_manager.cc:1397] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20250625 13:51:55.186725 1170 tablet_bootstrap.cc:654] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.176661 778 raft_consensus.cc:2466] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:55.188645 851 leader_election.cc:304] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.190068 1367 raft_consensus.cc:2802] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.190212 1367 raft_consensus.cc:695] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:55.190465 1367 consensus_queue.cc:237] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.191906 1171 raft_consensus.cc:357] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.192380 1171 raft_consensus.cc:383] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.192464 1171 raft_consensus.cc:738] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:51:55.192674 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36b151e200e1430c99ac194bc55881c1 has no consensus state
I20250625 13:51:55.192714 1171 consensus_queue.cc:260] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.193396 1171 ts_tablet_manager.cc:1428] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250625 13:51:55.193795 1171 tablet_bootstrap.cc:492] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.194829 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 07e281786e6e4a5d93b94b85b486655d from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.196102 1171 tablet_bootstrap.cc:654] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.196419 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 32221a413957450fa59ffcfbc91c51b9 with delete_type TABLET_DATA_DELETED (replaced by tablet bb1c7a85da844d34be805843bc60d350) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.197667 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2c63842cb66d43c19e91b079c23ad1df" } new_table_name: "default.Table_e1efe0a458494dc886f4dad68df5561f" modify_external_catalogs: true
I20250625 13:51:55.198240 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bdcf17e5612a44579ba27af546ca1def"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.198741 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdcf17e5612a44579ba27af546ca1def in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.199925 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 15750915ff7746ffa66509184be55dc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e0f2fa5d5614ad394e6e12ec62814d4) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.200428 884 tablet_service.cc:1515] Processing DeleteTablet for tablet d0cf3d979369419882a2ba2083933d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.202311 1168 ts_tablet_manager.cc:1918] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.202450 1168 log.cc:1199] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/872724cd95bf473cb40e5f8ed8b5bd55
I20250625 13:51:55.202862 1168 ts_tablet_manager.cc:1939] T 872724cd95bf473cb40e5f8ed8b5bd55 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.215797 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 122) took 1409 ms. Trace:
I20250625 13:51:55.216014 1170 tablet_bootstrap.cc:492] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.215929 1168 rpcz_store.cc:276] 0625 13:51:53.806038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.806107 (+ 69us) service_pool.cc:225] Handling call
0625 13:51:55.163149 (+1357042us) ts_tablet_manager.cc:1145] Deleting tablet 872724cd95bf473cb40e5f8ed8b5bd55
0625 13:51:55.163159 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.188378 (+ 25219us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.202153 (+ 13775us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.215757 (+ 13604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1356819}
I20250625 13:51:55.216179 1170 ts_tablet_manager.cc:1397] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250625 13:51:55.216369 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 124) took 1409 ms. Trace:
I20250625 13:51:55.216423 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 872724cd95bf473cb40e5f8ed8b5bd55 (table default.Table_c94d10947ecf49fba6f7608086ef35bc [id=263e68a7d9344016afdb8f4dc5b27b4f]) successfully deleted
I20250625 13:51:55.216465 1168 rpcz_store.cc:276] 0625 13:51:53.806933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.806976 (+ 43us) service_pool.cc:225] Handling call
0625 13:51:55.216220 (+1409244us) ts_tablet_manager.cc:1145] Deleting tablet 583ee2f2c66841939c93104efc36facb
0625 13:51:55.216231 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.216335 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409062}
I20250625 13:51:55.214823 1171 tablet_bootstrap.cc:492] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.217034 1170 raft_consensus.cc:357] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.217151 1171 ts_tablet_manager.cc:1397] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
W20250625 13:51:55.217162 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 583ee2f2c66841939c93104efc36facb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 583ee2f2c66841939c93104efc36facb already in progress: creating tablet
I20250625 13:51:55.217226 1170 raft_consensus.cc:383] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.217299 1170 raft_consensus.cc:738] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.217949 1171 raft_consensus.cc:357] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.218124 1171 raft_consensus.cc:383] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.218204 1171 raft_consensus.cc:738] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.218416 1171 consensus_queue.cc:260] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.218999 1171 ts_tablet_manager.cc:1428] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:51:55.219300 1171 tablet_bootstrap.cc:492] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.221599 1168 tablet_replica.cc:331] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.222339 1168 raft_consensus.cc:2241] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.222643 1168 pending_rounds.cc:70] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:51:55.222752 1168 pending_rounds.cc:77] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520569239339008 op_type: NO_OP noop_request { }
I20250625 13:51:55.222882 1168 raft_consensus.cc:2887] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:55.223031 1168 raft_consensus.cc:2270] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.223531 1167 ts_tablet_manager.cc:1918] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.223697 1168 ts_tablet_manager.cc:1905] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.223985 1167 log.cc:1199] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/39dde16d7f144ef880b52abf9c979a53
I20250625 13:51:55.224287 1170 consensus_queue.cc:260] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.225415 1167 ts_tablet_manager.cc:1939] T 39dde16d7f144ef880b52abf9c979a53 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.227916 1165 tablet_replica.cc:331] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:55.226909 1170 ts_tablet_manager.cc:1428] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 13:51:55.229256 1170 tablet_bootstrap.cc:492] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.231020 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 39dde16d7f144ef880b52abf9c979a53 (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]) successfully deleted
I20250625 13:51:55.234612 1171 tablet_bootstrap.cc:654] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.236037 1167 tablet_replica.cc:331] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:55.238806 1171 tablet_bootstrap.cc:492] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
W20250625 13:51:55.238940 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 19017cc372af43b2b5074a8990375ede because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19017cc372af43b2b5074a8990375ede already in progress: creating tablet
I20250625 13:51:55.239008 1171 ts_tablet_manager.cc:1397] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
W20250625 13:51:55.239238 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c911d9ecbded417daf70e85c7cbcd52e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c911d9ecbded417daf70e85c7cbcd52e already in progress: creating tablet
W20250625 13:51:55.239497 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f033bfa4925b4d1c971789eb77efd92c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f033bfa4925b4d1c971789eb77efd92c already in progress: creating tablet
W20250625 13:51:55.239707 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a21335a06ec04da29b3351554c75f03c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a21335a06ec04da29b3351554c75f03c already in progress: creating tablet
I20250625 13:51:55.239778 1167 raft_consensus.cc:2241] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.239986 1167 raft_consensus.cc:2270] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.239912 1171 raft_consensus.cc:357] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.240119 1171 raft_consensus.cc:383] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.240476 884 tablet_service.cc:1468] Processing CreateTablet for tablet abd6f9121a9d4b39adde1b880a875e23 (DEFAULT_TABLE table=default.Table_5a2f402bff584ef78d361f827bcd56f7 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.239527 1170 tablet_bootstrap.cc:654] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.240959 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd6f9121a9d4b39adde1b880a875e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.240918 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6d95a376304e464ab75e2e1e00e7a924 (DEFAULT_TABLE table=default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.241849 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d95a376304e464ab75e2e1e00e7a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.243849 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6d95a376304e464ab75e2e1e00e7a924 (DEFAULT_TABLE table=default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.244215 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d95a376304e464ab75e2e1e00e7a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.244249 1167 ts_tablet_manager.cc:1905] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.246273 1171 raft_consensus.cc:738] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.246577 1171 consensus_queue.cc:260] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.249994 591 catalog_manager.cc:5582] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } }
I20250625 13:51:55.250430 1018 tablet_service.cc:1468] Processing CreateTablet for tablet abd6f9121a9d4b39adde1b880a875e23 (DEFAULT_TABLE table=default.Table_5a2f402bff584ef78d361f827bcd56f7 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.250690 1171 ts_tablet_manager.cc:1428] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20250625 13:51:55.251148 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]). Expected version 1 got 0
I20250625 13:51:55.253798 1171 tablet_bootstrap.cc:492] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.256356 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd6f9121a9d4b39adde1b880a875e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.257508 1168 ts_tablet_manager.cc:1918] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.257639 1168 log.cc:1199] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a179c3698d424ec7b6d87bf65cb858cc
I20250625 13:51:55.258036 1168 ts_tablet_manager.cc:1939] T a179c3698d424ec7b6d87bf65cb858cc P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.258795 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 121) took 1453 ms. Trace:
I20250625 13:51:55.258906 1168 rpcz_store.cc:276] 0625 13:51:53.805494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.806236 (+ 742us) service_pool.cc:225] Handling call
0625 13:51:55.221527 (+1415291us) ts_tablet_manager.cc:1145] Deleting tablet a179c3698d424ec7b6d87bf65cb858cc
0625 13:51:55.221535 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.242971 (+ 21436us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.257372 (+ 14401us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.258764 (+ 1392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411515}
I20250625 13:51:55.259264 1168 tablet_replica.cc:331] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.263234 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet bf54376db991424b8de0e8f252fcb43f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.265144 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d5d3601056d240ef8a91362211b21f7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:55.265576 1168 raft_consensus.cc:2241] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.265777 1168 raft_consensus.cc:2270] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:55.265805 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5d3601056d240ef8a91362211b21f7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.266302 1168 ts_tablet_manager.cc:1905] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.260982 1171 tablet_bootstrap.cc:654] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.269454 1222 server_base.cc:1048] running on GCE node
I20250625 13:51:55.271787 1171 tablet_bootstrap.cc:492] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.271945 1171 ts_tablet_manager.cc:1397] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250625 13:51:55.272814 1171 raft_consensus.cc:357] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.273023 1171 raft_consensus.cc:383] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.273123 1171 raft_consensus.cc:738] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:51:55.273438 1370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:51:55.273404 1171 consensus_queue.cc:260] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.276278 1171 ts_tablet_manager.cc:1428] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:55.276618 1171 tablet_bootstrap.cc:492] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.278679 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8e5657d6a07a4399b5019e637f13c072
I20250625 13:51:55.284132 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_badc8f5a2d5040af96b9e7c9758c6417" } modify_external_catalogs: true
I20250625 13:51:55.284339 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e75b5fb51373451a94cb5ca59cdcefac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.284332 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_badc8f5a2d5040af96b9e7c9758c6417" } modify_external_catalogs: true
I20250625 13:51:55.286582 1171 tablet_bootstrap.cc:654] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.288136 1222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:51:55.289388 1168 ts_tablet_manager.cc:1918] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.289523 1168 log.cc:1199] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ca5dda8b27b34dafb20d8ed8afc58460
I20250625 13:51:55.289847 1168 ts_tablet_manager.cc:1939] T ca5dda8b27b34dafb20d8ed8afc58460 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.290627 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 123) took 1484 ms. Trace:
I20250625 13:51:55.291218 1222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:51:55.291254 1168 rpcz_store.cc:276] 0625 13:51:53.806496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.806562 (+ 66us) service_pool.cc:225] Handling call
0625 13:51:55.259204 (+1452642us) ts_tablet_manager.cc:1145] Deleting tablet ca5dda8b27b34dafb20d8ed8afc58460
0625 13:51:55.259213 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.283478 (+ 24265us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.289265 (+ 5787us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.290557 (+ 1292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448920}
I20250625 13:51:55.292436 1222 hybrid_clock.cc:648] HybridClock initialized: now 1750859515292363 us; error 59 us; skew 500 ppm
I20250625 13:51:55.292469 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 120) took 1487 ms. Trace:
I20250625 13:51:55.292991 1222 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:51:55.292948 1168 rpcz_store.cc:276] 0625 13:51:53.805083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:53.805531 (+ 448us) service_pool.cc:225] Handling call
0625 13:51:55.292309 (+1486778us) ts_tablet_manager.cc:1145] Deleting tablet 36da1da539e24f55978d516873cb0b27
0625 13:51:55.292319 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.292421 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1481759}
I20250625 13:51:55.293310 1174 raft_consensus.cc:491] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.293478 1174 raft_consensus.cc:513] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
W20250625 13:51:55.293725 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 36da1da539e24f55978d516873cb0b27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36da1da539e24f55978d516873cb0b27 already in progress: creating tablet
I20250625 13:51:55.294065 1168 tablet_replica.cc:331] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.294193 1174 leader_election.cc:290] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.294263 1168 raft_consensus.cc:2241] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.294451 1168 raft_consensus.cc:2270] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.294948 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.295158 916 raft_consensus.cc:2466] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:55.295769 983 leader_election.cc:304] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:55.296149 1174 raft_consensus.cc:2802] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.295002 1168 ts_tablet_manager.cc:1905] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.296269 1174 raft_consensus.cc:491] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.296368 1174 raft_consensus.cc:3058] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.297482 747 tablet_service.cc:1468] Processing CreateTablet for tablet 6d95a376304e464ab75e2e1e00e7a924 (DEFAULT_TABLE table=default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.298962 1174 raft_consensus.cc:513] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
W20250625 13:51:55.299401 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01ba67b500b646d7aa55315f7ee23ed6 consensus state has no leader
I20250625 13:51:55.300259 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:55.300439 916 raft_consensus.cc:3058] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.301230 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.301419 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d95a376304e464ab75e2e1e00e7a924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.303078 916 raft_consensus.cc:2466] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:55.303414 758 tablet_service.cc:1468] Processing CreateTablet for tablet abd6f9121a9d4b39adde1b880a875e23 (DEFAULT_TABLE table=default.Table_5a2f402bff584ef78d361f827bcd56f7 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.303560 983 leader_election.cc:304] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:55.303795 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd6f9121a9d4b39adde1b880a875e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.304477 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.305099 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.305445 1174 leader_election.cc:290] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.305583 1302 raft_consensus.cc:491] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.305657 1174 raft_consensus.cc:2802] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.305783 1174 raft_consensus.cc:695] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:55.305876 1302 raft_consensus.cc:513] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.306113 1174 consensus_queue.cc:237] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.306444 1302 leader_election.cc:290] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.305275 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.311468 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:55.312096 778 raft_consensus.cc:2466] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:55.316524 1170 tablet_bootstrap.cc:492] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.316721 1170 ts_tablet_manager.cc:1397] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.000s
I20250625 13:51:55.317623 1170 raft_consensus.cc:357] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.317993 1170 raft_consensus.cc:383] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.318094 1170 raft_consensus.cc:738] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.318353 1170 consensus_queue.cc:260] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.319213 1167 ts_tablet_manager.cc:1918] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20250625 13:51:55.319720 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 183296us
I20250625 13:51:55.320442 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0cf3d979369419882a2ba2083933d48" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.320787 916 raft_consensus.cc:2466] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:55.321070 1167 log.cc:1199] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/76f074156559406c875e65796e6b5ee6
I20250625 13:51:55.322023 1166 tablet_bootstrap.cc:492] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.322170 1166 ts_tablet_manager.cc:1397] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.177s user 0.004s sys 0.008s
I20250625 13:51:55.323010 1166 raft_consensus.cc:357] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.323027 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0cf3d979369419882a2ba2083933d48" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.323195 1166 raft_consensus.cc:383] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.323288 1166 raft_consensus.cc:738] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.323606 1166 consensus_queue.cc:260] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.324156 1166 ts_tablet_manager.cc:1428] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:51:55.324512 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_144bca3dd4db415b876a78a3b468e320"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.325109 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_144bca3dd4db415b876a78a3b468e320 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.323055 1167 ts_tablet_manager.cc:1939] T 76f074156559406c875e65796e6b5ee6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.326637 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]) successfully deleted
I20250625 13:51:55.326745 1167 tablet_replica.cc:331] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:55.326931 1167 raft_consensus.cc:2241] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:55.327060 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 67441795e22b445a90c88d76aa519ea5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67441795e22b445a90c88d76aa519ea5 already in progress: creating tablet
I20250625 13:51:55.327103 1167 raft_consensus.cc:2270] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.330087 1167 ts_tablet_manager.cc:1905] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.331458 1166 tablet_bootstrap.cc:492] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.334718 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.335011 916 raft_consensus.cc:2391] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c1aa07ab6bc4b0089e46951240a37c9 in current term 1: Already voted for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in this term.
I20250625 13:51:55.335793 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e75b5fb51373451a94cb5ca59cdcefac" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.336354 1303 raft_consensus.cc:491] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.336510 1303 raft_consensus.cc:513] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.335263 1166 tablet_bootstrap.cc:654] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.337123 1303 leader_election.cc:290] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.337246 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 04479dfb34584f0891c604ef72bf93b5
I20250625 13:51:55.338541 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 04479dfb34584f0891c604ef72bf93b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.338660 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 04479dfb34584f0891c604ef72bf93b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.338974 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32221a413957450fa59ffcfbc91c51b9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.339078 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 04479dfb34584f0891c604ef72bf93b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.339216 916 raft_consensus.cc:2466] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:55.340312 1171 tablet_bootstrap.cc:492] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.340548 1171 ts_tablet_manager.cc:1397] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.001s
I20250625 13:51:55.341100 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32221a413957450fa59ffcfbc91c51b9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:55.341646 711 leader_election.cc:343] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.341696 1171 raft_consensus.cc:357] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.341926 1171 raft_consensus.cc:383] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.342003 1171 raft_consensus.cc:738] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.342238 1171 consensus_queue.cc:260] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:55.343084 711 consensus_peers.cc:487] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0723e91650174f5badd74f7756c5a54e. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:51:55.344723 711 leader_election.cc:343] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.344766 1165 raft_consensus.cc:2241] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.345002 711 leader_election.cc:304] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.345780 1303 raft_consensus.cc:2802] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.345814 1170 ts_tablet_manager.cc:1428] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.029s user 0.007s sys 0.003s
I20250625 13:51:55.345891 1171 ts_tablet_manager.cc:1428] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:51:55.345888 1303 raft_consensus.cc:491] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.345989 1303 raft_consensus.cc:3058] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.346329 1170 tablet_bootstrap.cc:492] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.348150 1170 tablet_bootstrap.cc:654] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.348712 1171 tablet_bootstrap.cc:492] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.348661 1303 raft_consensus.cc:513] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.349289 1303 leader_election.cc:290] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.350564 1165 raft_consensus.cc:2270] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.352232 1165 ts_tablet_manager.cc:1905] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.355726 711 leader_election.cc:304] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:55.356020 1303 consensus_queue.cc:1035] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:55.356170 1171 tablet_bootstrap.cc:654] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.356567 1303 raft_consensus.cc:3058] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.357198 1222 webserver.cc:469] Webserver started at http://127.0.104.62:44303/ using document root <none> and password file <none>
I20250625 13:51:55.357990 1166 tablet_bootstrap.cc:492] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.358196 1166 ts_tablet_manager.cc:1397] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250625 13:51:55.359103 1166 raft_consensus.cc:357] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.359330 1166 raft_consensus.cc:383] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.359349 1222 fs_manager.cc:362] Metadata directory not provided
I20250625 13:51:55.359431 1166 raft_consensus.cc:738] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.359524 1222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:51:55.359619 1303 raft_consensus.cc:2747] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:55.359730 1166 consensus_queue.cc:260] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.360148 711 leader_election.cc:304] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.360244 1166 ts_tablet_manager.cc:1428] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:51:55.360613 1166 tablet_bootstrap.cc:492] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.361394 1165 ts_tablet_manager.cc:1918] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.361955 1165 log.cc:1199] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/0723e91650174f5badd74f7756c5a54e
I20250625 13:51:55.364012 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 07e281786e6e4a5d93b94b85b486655d
I20250625 13:51:55.364488 592 catalog_manager.cc:6652] ReplaceTablet: tablet 07e281786e6e4a5d93b94b85b486655d deleted and replaced by tablet be3313ec5dfb41b68662d3ff4c8c94f9
I20250625 13:51:55.364867 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d9dd2f12f8bb4f6cba435c892dac832e" } modify_external_catalogs: true
I20250625 13:51:55.365559 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d9dd2f12f8bb4f6cba435c892dac832e" } modify_external_catalogs: true
I20250625 13:51:55.366892 1167 ts_tablet_manager.cc:1918] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.366935 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6771d8e180fe48569a0e016db0cc9b51
I20250625 13:51:55.367031 1167 log.cc:1199] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ca5dda8b27b34dafb20d8ed8afc58460
I20250625 13:51:55.367156 1168 ts_tablet_manager.cc:1918] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.367262 1168 log.cc:1199] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/921163dc5f454c9a97e72b005f9dba19
I20250625 13:51:55.367379 588 catalog_manager.cc:6652] ReplaceTablet: tablet 6771d8e180fe48569a0e016db0cc9b51 deleted and replaced by tablet 86af0ea6648045a1ac4ba903d20849b7
I20250625 13:51:55.367725 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f728647d61a4414ca11ff2b890235012"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.368276 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f728647d61a4414ca11ff2b890235012 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.369515 1303 raft_consensus.cc:2802] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.369642 1303 raft_consensus.cc:491] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.369704 1165 ts_tablet_manager.cc:1939] T 0723e91650174f5badd74f7756c5a54e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:55.369773 1303 raft_consensus.cc:3058] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.370558 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 141) took 1272 ms. Trace:
I20250625 13:51:55.370747 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e281786e6e4a5d93b94b85b486655d with delete_type TABLET_DATA_DELETED (replaced by tablet be3313ec5dfb41b68662d3ff4c8c94f9) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.370689 1165 rpcz_store.cc:276] 0625 13:51:54.098447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.114614 (+ 16167us) service_pool.cc:225] Handling call
0625 13:51:55.155583 (+1040969us) ts_tablet_manager.cc:1145] Deleting tablet 0723e91650174f5badd74f7756c5a54e
0625 13:51:55.166168 (+ 10585us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.345181 (+179013us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x6160000f1c54. stack: 00007f4161830336 00007f41618300f0 00007f41608263ad 0000000000971004 0000000000970fcd 000000000097099a 00007f416d290000 00007f416e5a2093 00007f417107f137 00007f41710b11c4 00007f4171092c20 00007f41710929ad 0000000000a7573c 00007f41618759df 00007f4161878cab 00007f4161878b28
0625 13:51:55.356795 (+ 11614us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.361238 (+ 4443us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.370512 (+ 9274us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78939,"spinlock_wait_cycles":86479232,"tablet-delete.queue_time_us":1002483}
I20250625 13:51:55.370971 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 with delete_type TABLET_DATA_DELETED (replaced by tablet 86af0ea6648045a1ac4ba903d20849b7) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.371217 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 146) took 1199 ms. Trace:
I20250625 13:51:55.371364 1165 rpcz_store.cc:276] 0625 13:51:54.171978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.172032 (+ 54us) service_pool.cc:225] Handling call
0625 13:51:55.371038 (+1199006us) ts_tablet_manager.cc:1145] Deleting tablet 799f2edd960d45c1ace9195283a042e2
0625 13:51:55.371048 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.371175 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198760}
I20250625 13:51:55.371448 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0cf3d979369419882a2ba2083933d48" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:55.371646 916 raft_consensus.cc:3058] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.371693 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 147) took 1199 ms. Trace:
I20250625 13:51:55.372160 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
W20250625 13:51:55.372417 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 799f2edd960d45c1ace9195283a042e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 799f2edd960d45c1ace9195283a042e2 already in progress: creating tablet
I20250625 13:51:55.372570 1303 raft_consensus.cc:513] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.372702 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e281786e6e4a5d93b94b85b486655d with delete_type TABLET_DATA_DELETED (replaced by tablet be3313ec5dfb41b68662d3ff4c8c94f9) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.372965 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e281786e6e4a5d93b94b85b486655d with delete_type TABLET_DATA_DELETED (replaced by tablet be3313ec5dfb41b68662d3ff4c8c94f9) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.372990 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 with delete_type TABLET_DATA_DELETED (replaced by tablet 86af0ea6648045a1ac4ba903d20849b7) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.373404 1303 leader_election.cc:290] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.373461 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 with delete_type TABLET_DATA_DELETED (replaced by tablet 86af0ea6648045a1ac4ba903d20849b7) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.371795 1165 rpcz_store.cc:276] 0625 13:51:54.172118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.172163 (+ 45us) service_pool.cc:225] Handling call
0625 13:51:55.371573 (+1199410us) ts_tablet_manager.cc:1145] Deleting tablet 4b549bc8cead4af4a72f41f51655602c
0625 13:51:55.371582 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.371657 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199187}
W20250625 13:51:55.374753 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4b549bc8cead4af4a72f41f51655602c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b549bc8cead4af4a72f41f51655602c already in progress: creating tablet
I20250625 13:51:55.375162 1166 tablet_bootstrap.cc:654] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.375193 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 148) took 1202 ms. Trace:
I20250625 13:51:55.375456 1165 rpcz_store.cc:276] 0625 13:51:54.172267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.172423 (+ 156us) service_pool.cc:225] Handling call
0625 13:51:55.375043 (+1202620us) ts_tablet_manager.cc:1145] Deleting tablet b7641869a0bb463698244ff445d87a1f
0625 13:51:55.375054 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.375156 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202396}
W20250625 13:51:55.376037 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet b7641869a0bb463698244ff445d87a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7641869a0bb463698244ff445d87a1f already in progress: creating tablet
I20250625 13:51:55.376471 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 145) took 1204 ms. Trace:
I20250625 13:51:55.376633 1165 rpcz_store.cc:276] 0625 13:51:54.171858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.178208 (+ 6350us) service_pool.cc:225] Handling call
0625 13:51:55.376325 (+1198117us) ts_tablet_manager.cc:1145] Deleting tablet 2c1a98dc40684954a602567cf3476879
0625 13:51:55.376336 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.376426 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197781}
I20250625 13:51:55.374380 916 raft_consensus.cc:2466] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
W20250625 13:51:55.377111 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2c1a98dc40684954a602567cf3476879 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c1a98dc40684954a602567cf3476879 already in progress: creating tablet
I20250625 13:51:55.377502 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 153) took 1104 ms. Trace:
I20250625 13:51:55.378163 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0cf3d979369419882a2ba2083933d48" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:55.378161 1165 rpcz_store.cc:276] 0625 13:51:54.273052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.273495 (+ 443us) service_pool.cc:225] Handling call
0625 13:51:55.377345 (+1103850us) ts_tablet_manager.cc:1145] Deleting tablet 5786bb7716ef44689ef9862fc3ce17e9
0625 13:51:55.377353 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.377464 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103612}
W20250625 13:51:55.378563 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5786bb7716ef44689ef9862fc3ce17e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5786bb7716ef44689ef9862fc3ce17e9 already in progress: creating tablet
I20250625 13:51:55.378862 1171 tablet_bootstrap.cc:492] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.378916 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 155) took 1056 ms. Trace:
I20250625 13:51:55.379019 1171 ts_tablet_manager.cc:1397] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250625 13:51:55.379020 1165 rpcz_store.cc:276] 0625 13:51:54.322248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.322303 (+ 55us) service_pool.cc:225] Handling call
0625 13:51:55.378787 (+1056484us) ts_tablet_manager.cc:1145] Deleting tablet 93f449c80b604a6498f6fd1d1ba4770a
0625 13:51:55.378794 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.378886 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1056170}
W20250625 13:51:55.379420 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 93f449c80b604a6498f6fd1d1ba4770a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93f449c80b604a6498f6fd1d1ba4770a already in progress: creating tablet
I20250625 13:51:55.380028 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 154) took 1058 ms. Trace:
I20250625 13:51:55.379948 1171 raft_consensus.cc:357] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.380136 1171 raft_consensus.cc:383] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.380245 1171 raft_consensus.cc:738] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.380486 1171 consensus_queue.cc:260] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.380127 1165 rpcz_store.cc:276] 0625 13:51:54.321945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.351813 (+ 29868us) service_pool.cc:225] Handling call
0625 13:51:55.379877 (+1028064us) ts_tablet_manager.cc:1145] Deleting tablet 8149aea0ff7646d3b94c9159f56e48ef
0625 13:51:55.379887 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.379998 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027728}
I20250625 13:51:55.381057 1171 ts_tablet_manager.cc:1428] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 13:51:55.381316 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8149aea0ff7646d3b94c9159f56e48ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8149aea0ff7646d3b94c9159f56e48ef already in progress: creating tablet
I20250625 13:51:55.381362 1171 tablet_bootstrap.cc:492] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.383105 1171 tablet_bootstrap.cc:654] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.383160 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 85c72f9ae92a4537979fb479eb4467c4
I20250625 13:51:55.383654 1165 tablet_replica.cc:331] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:55.383872 1165 raft_consensus.cc:2241] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.384028 1165 raft_consensus.cc:2270] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.385047 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32221a413957450fa59ffcfbc91c51b9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:55.385239 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.385588 1170 tablet_bootstrap.cc:492] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.385742 1170 ts_tablet_manager.cc:1397] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.001s
W20250625 13:51:55.386374 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 19017cc372af43b2b5074a8990375ede because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19017cc372af43b2b5074a8990375ede already in progress: creating tablet
W20250625 13:51:55.386554 711 leader_election.cc:343] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.386505 1170 raft_consensus.cc:357] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
W20250625 13:51:55.386617 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c911d9ecbded417daf70e85c7cbcd52e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c911d9ecbded417daf70e85c7cbcd52e already in progress: creating tablet
I20250625 13:51:55.386687 1170 raft_consensus.cc:383] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.386747 1170 raft_consensus.cc:738] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.386883 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32221a413957450fa59ffcfbc91c51b9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:55.386989 1170 consensus_queue.cc:260] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.387094 915 raft_consensus.cc:3058] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.387426 1170 ts_tablet_manager.cc:1428] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 13:51:55.386797 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f033bfa4925b4d1c971789eb77efd92c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f033bfa4925b4d1c971789eb77efd92c already in progress: creating tablet
I20250625 13:51:55.387704 1170 tablet_bootstrap.cc:492] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:51:55.386776 711 leader_election.cc:343] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.389636 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.390161 1303 consensus_queue.cc:1035] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:55.390712 711 leader_election.cc:304] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.391207 1303 raft_consensus.cc:2802] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.391299 1165 ts_tablet_manager.cc:1905] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.391386 1303 raft_consensus.cc:695] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:55.391417 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.391723 1303 consensus_queue.cc:237] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.392033 1168 ts_tablet_manager.cc:1939] T 921163dc5f454c9a97e72b005f9dba19 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.392831 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 125) took 1371 ms. Trace:
I20250625 13:51:55.393002 1168 rpcz_store.cc:276] 0625 13:51:54.021621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.021680 (+ 59us) service_pool.cc:225] Handling call
0625 13:51:55.293986 (+1272306us) ts_tablet_manager.cc:1145] Deleting tablet 921163dc5f454c9a97e72b005f9dba19
0625 13:51:55.293995 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.363428 (+ 69433us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.367039 (+ 3611us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.392781 (+ 25742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1272069}
I20250625 13:51:55.393477 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 921163dc5f454c9a97e72b005f9dba19 (table default.Table_b13c5d5237f044c9bad038533aecc142 [id=96df25b43f664e9ebc958ff801d229b5]) successfully deleted
I20250625 13:51:55.396188 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4a201c5623af4b8a95f24f43319311f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.396847 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a201c5623af4b8a95f24f43319311f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.399241 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b11b3157dd764f229831c3b8417460c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 85c72f9ae92a4537979fb479eb4467c4) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.400241 1168 tablet_replica.cc:331] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.400265 915 raft_consensus.cc:2466] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:55.402940 1168 raft_consensus.cc:2241] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.403169 1168 raft_consensus.cc:2270] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.406955 1165 ts_tablet_manager.cc:1918] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.407112 1165 log.cc:1199] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a21335a06ec04da29b3351554c75f03c
I20250625 13:51:55.407162 1166 tablet_bootstrap.cc:492] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.407320 1166 ts_tablet_manager.cc:1397] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.001s
I20250625 13:51:55.407538 1165 ts_tablet_manager.cc:1939] T a21335a06ec04da29b3351554c75f03c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:55.409582 1168 ts_tablet_manager.cc:1905] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.412293 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a21335a06ec04da29b3351554c75f03c (table default.Table_1329da083f314d9d8fa3ac2e851646bf [id=3286dc94d1ba4645beb4a2a54bec8c8a]) successfully deleted
I20250625 13:51:55.408191 1166 raft_consensus.cc:357] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.413829 1166 raft_consensus.cc:383] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.413944 1368 raft_consensus.cc:491] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.414013 1166 raft_consensus.cc:738] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.414251 1166 consensus_queue.cc:260] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:55.414330 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 11711546f0e14f9a9eabf003f49a69b7 (table default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 11711546f0e14f9a9eabf003f49a69b7 consensus state has no leader
I20250625 13:51:55.414098 1368 raft_consensus.cc:513] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.415043 1368 leader_election.cc:290] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.411219 711 leader_election.cc:304] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.412632 1165 tablet_replica.cc:331] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:55.415354 1165 raft_consensus.cc:2241] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.415484 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f00e72e103f443e886f0df690cff4b6f
I20250625 13:51:55.415544 1165 raft_consensus.cc:2270] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.415973 595 catalog_manager.cc:6652] ReplaceTablet: tablet f00e72e103f443e886f0df690cff4b6f deleted and replaced by tablet a0e81d4c904248eba858462bcf31c5d9
I20250625 13:51:55.415210 1166 ts_tablet_manager.cc:1428] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 13:51:55.416297 1166 tablet_bootstrap.cc:492] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.416258 758 tablet_service.cc:1515] Processing DeleteTablet for tablet f00e72e103f443e886f0df690cff4b6f with delete_type TABLET_DATA_DELETED (replaced by tablet a0e81d4c904248eba858462bcf31c5d9) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.416586 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "921163dc5f454c9a97e72b005f9dba19" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.416728 1165 ts_tablet_manager.cc:1905] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:55.417737 711 leader_election.cc:343] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.417863 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 9a31f10c8ca745b7ad635c622a265450 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.415364 1368 raft_consensus.cc:2802] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.418251 1166 tablet_bootstrap.cc:654] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.419011 1368 raft_consensus.cc:695] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:55.419365 1368 consensus_queue.cc:237] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.420243 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f00e72e103f443e886f0df690cff4b6f with delete_type TABLET_DATA_DELETED (replaced by tablet a0e81d4c904248eba858462bcf31c5d9) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.420506 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet ce84f74470ad428abb2113efa170ca24 with delete_type TABLET_DATA_DELETED (replaced by tablet 01ba67b500b646d7aa55315f7ee23ed6) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.440615 1165 ts_tablet_manager.cc:1918] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.440774 1165 log.cc:1199] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/bf54376db991424b8de0e8f252fcb43f
I20250625 13:51:55.441169 1165 ts_tablet_manager.cc:1939] T bf54376db991424b8de0e8f252fcb43f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:55.442163 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet bf54376db991424b8de0e8f252fcb43f (table default.Table_719d67c948c8495ba1f5c2001c020ce5 [id=8504f82376b440a9a3ab8c9bf1741cfe]) successfully deleted
I20250625 13:51:55.442325 1170 tablet_bootstrap.cc:654] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.448410 1167 ts_tablet_manager.cc:1939] T ca5dda8b27b34dafb20d8ed8afc58460 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.449777 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ca5dda8b27b34dafb20d8ed8afc58460 (table default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]) successfully deleted
I20250625 13:51:55.450451 1307 raft_consensus.cc:491] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.450568 1367 raft_consensus.cc:491] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.450840 1307 raft_consensus.cc:513] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:55.451023 711 leader_election.cc:343] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 921163dc5f454c9a97e72b005f9dba19
I20250625 13:51:55.451167 711 leader_election.cc:304] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:55.451277 1171 tablet_bootstrap.cc:492] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.451448 1303 raft_consensus.cc:2747] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:55.451941 1165 tablet_replica.cc:331] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:55.452211 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a7b72c2c464421eacfcf64008c3aea8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.452221 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a7b72c2c464421eacfcf64008c3aea8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.452620 1165 raft_consensus.cc:2241] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.452879 1165 pending_rounds.cc:70] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:55.452981 1165 pending_rounds.cc:77] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520573643923456 op_type: NO_OP noop_request { }
I20250625 13:51:55.453107 1165 raft_consensus.cc:2887] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:55.453241 1165 raft_consensus.cc:2270] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.453452 1307 leader_election.cc:290] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.453660 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f00e72e103f443e886f0df690cff4b6f with delete_type TABLET_DATA_DELETED (replaced by tablet a0e81d4c904248eba858462bcf31c5d9) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.453478 1171 ts_tablet_manager.cc:1397] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.003s
I20250625 13:51:55.454047 1367 raft_consensus.cc:513] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.454675 1165 ts_tablet_manager.cc:1905] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.454664 1367 leader_election.cc:290] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.454711 1171 raft_consensus.cc:357] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.454798 1166 tablet_bootstrap.cc:492] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.454945 1166 ts_tablet_manager.cc:1397] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.004s
I20250625 13:51:55.455500 1170 tablet_bootstrap.cc:492] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.455691 1166 raft_consensus.cc:357] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.455642 1170 ts_tablet_manager.cc:1397] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.002s
I20250625 13:51:55.456097 1171 raft_consensus.cc:383] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.455876 1166 raft_consensus.cc:383] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.456202 1171 raft_consensus.cc:738] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.456264 1166 raft_consensus.cc:738] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:51:55.456575 849 leader_election.cc:343] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.456502 1166 consensus_queue.cc:260] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.457007 1166 ts_tablet_manager.cc:1428] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:55.457114 1170 raft_consensus.cc:357] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.457320 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1a98dc40684954a602567cf3476879" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:55.457921 851 leader_election.cc:343] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.457890 1171 consensus_queue.cc:260] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.458143 851 leader_election.cc:304] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:55.458436 1171 ts_tablet_manager.cc:1428] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 13:51:55.457297 1170 raft_consensus.cc:383] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.458801 1170 raft_consensus.cc:738] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.459120 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1a98dc40684954a602567cf3476879" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.459080 1170 consensus_queue.cc:260] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.459615 1170 ts_tablet_manager.cc:1428] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:55.459981 1171 tablet_bootstrap.cc:492] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.460879 1170 tablet_bootstrap.cc:492] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.461115 1307 raft_consensus.cc:2747] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:55.462181 1166 tablet_bootstrap.cc:492] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.462452 1171 tablet_bootstrap.cc:654] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.462581 1168 ts_tablet_manager.cc:1918] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:51:55.463127 849 leader_election.cc:343] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:55.463747 851 leader_election.cc:343] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.463963 851 leader_election.cc:304] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:55.464368 1307 raft_consensus.cc:2747] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:55.464692 1168 log.cc:1199] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/15750915ff7746ffa66509184be55dc4
I20250625 13:51:55.465059 1168 ts_tablet_manager.cc:1939] T 15750915ff7746ffa66509184be55dc4 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.465519 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 8b7062c34fd143bf86b743055a27db8c from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.465970 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c666ff2d6b9b4ed4a18e370826c143fc
I20250625 13:51:55.465975 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 126) took 1443 ms. Trace:
I20250625 13:51:55.466068 1168 rpcz_store.cc:276] 0625 13:51:54.022077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.022122 (+ 45us) service_pool.cc:225] Handling call
0625 13:51:55.394428 (+1372306us) ts_tablet_manager.cc:1145] Deleting tablet 15750915ff7746ffa66509184be55dc4
0625 13:51:55.394439 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.412838 (+ 18399us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.462454 (+ 49616us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.465920 (+ 3466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372121}
I20250625 13:51:55.467250 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 127) took 1444 ms. Trace:
I20250625 13:51:55.467419 1168 rpcz_store.cc:276] 0625 13:51:54.022441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.022639 (+ 198us) service_pool.cc:225] Handling call
0625 13:51:55.466987 (+1444348us) ts_tablet_manager.cc:1145] Deleting tablet 87327f74beb845768bbfd5ae6d2bcf88
0625 13:51:55.466997 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.467190 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444173}
I20250625 13:51:55.467983 1168 tablet_replica.cc:331] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.468161 1170 tablet_bootstrap.cc:654] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.468365 1168 raft_consensus.cc:2241] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.468724 1168 pending_rounds.cc:70] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:51:55.468708 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 15750915ff7746ffa66509184be55dc4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]) successfully deleted
I20250625 13:51:55.468920 1168 pending_rounds.cc:77] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520573779603456 op_type: NO_OP noop_request { }
W20250625 13:51:55.469040 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 87327f74beb845768bbfd5ae6d2bcf88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87327f74beb845768bbfd5ae6d2bcf88 already in progress: creating tablet
W20250625 13:51:55.469228 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d0c5bea3b6214fa8af2792c5e9c411d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0c5bea3b6214fa8af2792c5e9c411d9 already in progress: creating tablet
W20250625 13:51:55.469403 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 15e58fbec94e4ff9b7ac9830b576fe7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15e58fbec94e4ff9b7ac9830b576fe7f already in progress: creating tablet
W20250625 13:51:55.469560 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2cd5303e674246ada7d36574b535b333 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd5303e674246ada7d36574b535b333 already in progress: creating tablet
I20250625 13:51:55.469501 1168 raft_consensus.cc:2887] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20250625 13:51:55.469751 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 14a12a5453e647b9946ca19e0ffb14ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14a12a5453e647b9946ca19e0ffb14ee already in progress: creating tablet
I20250625 13:51:55.469871 1167 tablet_replica.cc:331] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:51:55.469907 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 389d81329b8d430a9098b0630612fdcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 389d81329b8d430a9098b0630612fdcb already in progress: creating tablet
I20250625 13:51:55.469906 1168 raft_consensus.cc:2270] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.470908 1167 raft_consensus.cc:2241] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.471226 1168 ts_tablet_manager.cc:1905] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.471498 1167 raft_consensus.cc:2270] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.471908 1166 tablet_bootstrap.cc:654] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.473227 1167 ts_tablet_manager.cc:1905] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.476369 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.476696 884 tablet_service.cc:1515] Processing DeleteTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.476924 895 tablet_service.cc:1515] Processing DeleteTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.477370 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.477766 1170 tablet_bootstrap.cc:492] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.477955 1170 ts_tablet_manager.cc:1397] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250625 13:51:55.478721 758 tablet_service.cc:1515] Processing DeleteTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.478775 1170 raft_consensus.cc:357] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.478940 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.478997 1170 raft_consensus.cc:383] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.479070 1170 raft_consensus.cc:738] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.479326 1170 consensus_queue.cc:260] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.479879 1170 ts_tablet_manager.cc:1428] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:51:55.480746 1170 tablet_bootstrap.cc:492] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.482261 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_10094a18787042588bd628e824da998f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.482895 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10094a18787042588bd628e824da998f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.484424 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c3f37af0b4094d84909408f68b8b1314" } new_table_name: "default.Table_14940f9e4a2645aab4d9b490b94171b3" modify_external_catalogs: true
I20250625 13:51:55.484644 1165 ts_tablet_manager.cc:1918] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.484803 1165 log.cc:1199] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/39dde16d7f144ef880b52abf9c979a53
I20250625 13:51:55.485230 1165 ts_tablet_manager.cc:1939] T 39dde16d7f144ef880b52abf9c979a53 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:55.485513 1171 tablet_bootstrap.cc:492] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.485661 1171 ts_tablet_manager.cc:1397] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.003s
I20250625 13:51:55.485698 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7fe053e294fe4aceb24919e9e2bddf65
I20250625 13:51:55.486191 1165 tablet_replica.cc:331] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:55.486330 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cc6c89513ac9408bbfb6387793e10d33" } new_table_name: "default.Table_1f2f5fdc47f0401bb672e1ae9c82473a" modify_external_catalogs: true
I20250625 13:51:55.486397 1165 raft_consensus.cc:2241] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.486553 1165 raft_consensus.cc:2270] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.486479 1171 raft_consensus.cc:357] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.486665 1171 raft_consensus.cc:383] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.486745 1171 raft_consensus.cc:738] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.486974 1171 consensus_queue.cc:260] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.487569 1171 ts_tablet_manager.cc:1428] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:51:55.487849 1171 tablet_bootstrap.cc:492] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.490079 1171 tablet_bootstrap.cc:654] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.490347 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7a4c3ced3be545189e17b6f63f336f65" } new_table_name: "default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae" modify_external_catalogs: true
I20250625 13:51:55.490417 1165 ts_tablet_manager.cc:1905] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.491550 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6264bd28054f4c52a39e6f6a6a4199bc" } new_table_name: "default.Table_fa5420c9509b49d5a11c3b0a059f9f9c" modify_external_catalogs: true
I20250625 13:51:55.492954 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fe053e294fe4aceb24919e9e2bddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.493182 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 39dde16d7f144ef880b52abf9c979a53 (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]) successfully deleted
I20250625 13:51:55.494390 1167 ts_tablet_manager.cc:1918] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.496531 1167 log.cc:1199] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/15750915ff7746ffa66509184be55dc4
I20250625 13:51:55.497126 1167 ts_tablet_manager.cc:1939] T 15750915ff7746ffa66509184be55dc4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.495895 1170 tablet_bootstrap.cc:654] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.501114 1047 raft_consensus.cc:3058] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.504010 1171 tablet_bootstrap.cc:492] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.504165 1171 ts_tablet_manager.cc:1397] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20250625 13:51:55.505081 1171 raft_consensus.cc:357] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.505234 1307 consensus_queue.cc:1035] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:55.505511 1167 tablet_replica.cc:331] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:55.505301 1171 raft_consensus.cc:383] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.505738 1171 raft_consensus.cc:738] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.505999 1171 consensus_queue.cc:260] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:55.506635 849 consensus_peers.cc:487] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:55.507876 1171 ts_tablet_manager.cc:1428] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:51:55.508875 1171 tablet_bootstrap.cc:492] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.511099 1171 tablet_bootstrap.cc:654] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.512323 1167 raft_consensus.cc:2241] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.512285 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 15750915ff7746ffa66509184be55dc4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]) successfully deleted
I20250625 13:51:55.512557 1167 raft_consensus.cc:2270] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.512754 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fe053e294fe4aceb24919e9e2bddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.513129 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fe053e294fe4aceb24919e9e2bddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.513352 1167 ts_tablet_manager.cc:1905] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.516726 1170 tablet_bootstrap.cc:492] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.516912 1170 ts_tablet_manager.cc:1397] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.002s
I20250625 13:51:55.517771 1170 raft_consensus.cc:357] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.517957 1170 raft_consensus.cc:383] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.518056 1170 raft_consensus.cc:738] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.518333 1170 consensus_queue.cc:260] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.518848 1170 ts_tablet_manager.cc:1428] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:55.519146 1170 tablet_bootstrap.cc:492] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.519569 1222 fs_manager.cc:714] Time spent opening directory manager: real 0.117s user 0.005s sys 0.000s
I20250625 13:51:55.520752 1170 tablet_bootstrap.cc:654] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.527531 1168 ts_tablet_manager.cc:1918] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.527711 1168 log.cc:1199] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ce84f74470ad428abb2113efa170ca24
I20250625 13:51:55.528084 1168 ts_tablet_manager.cc:1939] T ce84f74470ad428abb2113efa170ca24 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.528117 1166 tablet_bootstrap.cc:492] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.528288 1166 ts_tablet_manager.cc:1397] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.007s
I20250625 13:51:55.528806 1167 ts_tablet_manager.cc:1918] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.528944 1167 log.cc:1199] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/bf54376db991424b8de0e8f252fcb43f
I20250625 13:51:55.529300 1167 ts_tablet_manager.cc:1939] T bf54376db991424b8de0e8f252fcb43f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.529258 1166 raft_consensus.cc:357] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.529487 1166 raft_consensus.cc:383] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.529595 1166 raft_consensus.cc:738] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.530079 1166 consensus_queue.cc:260] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.530367 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet bf54376db991424b8de0e8f252fcb43f (table default.Table_719d67c948c8495ba1f5c2001c020ce5 [id=8504f82376b440a9a3ab8c9bf1741cfe]) successfully deleted
I20250625 13:51:55.533635 1166 ts_tablet_manager.cc:1428] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250625 13:51:55.534451 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 128) took 1511 ms. Trace:
I20250625 13:51:55.534590 1168 rpcz_store.cc:276] 0625 13:51:54.022997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.023041 (+ 44us) service_pool.cc:225] Handling call
0625 13:51:55.467882 (+1444841us) ts_tablet_manager.cc:1145] Deleting tablet ce84f74470ad428abb2113efa170ca24
0625 13:51:55.467894 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.474788 (+ 6894us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.527410 (+ 52622us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.534377 (+ 6967us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444639}
I20250625 13:51:55.535187 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet ce84f74470ad428abb2113efa170ca24 (table default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]) successfully deleted
I20250625 13:51:55.535661 1168 tablet_replica.cc:331] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.535903 1168 raft_consensus.cc:2241] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.536595 1166 tablet_bootstrap.cc:492] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.536870 1167 tablet_replica.cc:331] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:55.537075 1167 raft_consensus.cc:2241] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.537313 1167 pending_rounds.cc:70] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:51:55.537401 1167 pending_rounds.cc:77] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520574699376640 op_type: NO_OP noop_request { }
I20250625 13:51:55.537505 1167 raft_consensus.cc:2887] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:55.537616 1167 raft_consensus.cc:2270] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.537729 1165 ts_tablet_manager.cc:1918] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.537976 1165 log.cc:1199] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/e75b5fb51373451a94cb5ca59cdcefac
I20250625 13:51:55.538434 1165 ts_tablet_manager.cc:1939] T e75b5fb51373451a94cb5ca59cdcefac P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:55.540223 1165 tablet_replica.cc:331] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:55.540477 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet e75b5fb51373451a94cb5ca59cdcefac (table default.Table_e75519f3b879406ea7bb8f515825f274 [id=ffb244ee24124e8a83db14b5747abed7]) successfully deleted
I20250625 13:51:55.541481 1167 ts_tablet_manager.cc:1905] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.541846 1168 raft_consensus.cc:2270] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.542210 1307 raft_consensus.cc:491] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.542362 1307 raft_consensus.cc:513] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.543135 1307 leader_election.cc:290] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.544719 1171 tablet_bootstrap.cc:492] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.544883 1171 ts_tablet_manager.cc:1397] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.006s
I20250625 13:51:55.545639 1168 ts_tablet_manager.cc:1905] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.546525 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0f2fa5d5614ad394e6e12ec62814d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.545826 1171 raft_consensus.cc:357] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.546886 1171 raft_consensus.cc:383] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.547009 1171 raft_consensus.cc:738] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.546947 1412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:51:55.547269 1171 consensus_queue.cc:260] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.548626 1171 ts_tablet_manager.cc:1428] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:55.549748 1307 raft_consensus.cc:491] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:51:55.549804 849 leader_election.cc:343] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.549893 1307 raft_consensus.cc:513] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.550457 1307 leader_election.cc:290] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.550765 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13620f6ca65342b4ad7f1ba4b437bde3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:55.551256 849 leader_election.cc:343] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.551481 1171 tablet_bootstrap.cc:492] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.553151 1171 tablet_bootstrap.cc:654] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.555472 1170 tablet_bootstrap.cc:492] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.555806 1170 ts_tablet_manager.cc:1397] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.005s
I20250625 13:51:55.556715 1307 raft_consensus.cc:491] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.556914 1307 raft_consensus.cc:513] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.557551 1307 leader_election.cc:290] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.558404 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5a6951aba84f5fa8f6d76a2b7fcf2d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:55.559022 849 leader_election.cc:343] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.561200 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9a31f10c8ca745b7ad635c622a265450
I20250625 13:51:55.561190 1170 raft_consensus.cc:357] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.561388 1170 raft_consensus.cc:383] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.561460 1170 raft_consensus.cc:738] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.561689 595 catalog_manager.cc:6652] ReplaceTablet: tablet 9a31f10c8ca745b7ad635c622a265450 deleted and replaced by tablet df1ed922750f45eab02d1fc0c26af75e
I20250625 13:51:55.561689 1170 consensus_queue.cc:260] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.562072 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1310c8c1f7634350b0a714f5150ef5a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:55.562242 1170 ts_tablet_manager.cc:1428] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 13:51:55.562573 1170 tablet_bootstrap.cc:492] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:51:55.562667 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1310c8c1f7634350b0a714f5150ef5a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.562952 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a31f10c8ca745b7ad635c622a265450 with delete_type TABLET_DATA_DELETED (replaced by tablet df1ed922750f45eab02d1fc0c26af75e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.563228 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a31f10c8ca745b7ad635c622a265450 with delete_type TABLET_DATA_DELETED (replaced by tablet df1ed922750f45eab02d1fc0c26af75e) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.564393 1170 tablet_bootstrap.cc:654] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.565534 1222 fs_manager.cc:730] Time spent opening block manager: real 0.041s user 0.008s sys 0.001s
I20250625 13:51:55.565737 1222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
uuid: "3fec9d8a79054a5991ae90d8a3101e32"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
I20250625 13:51:55.566812 1222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:51:55.568164 1166 tablet_bootstrap.cc:654] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.569862 1171 tablet_bootstrap.cc:492] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.569900 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0f2fa5d5614ad394e6e12ec62814d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.570030 1171 ts_tablet_manager.cc:1397] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250625 13:51:55.570098 778 raft_consensus.cc:2466] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:55.570186 1165 raft_consensus.cc:2241] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.570734 851 leader_election.cc:304] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.571147 1307 raft_consensus.cc:2802] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.571027 1171 raft_consensus.cc:357] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.571239 1307 raft_consensus.cc:491] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.571837 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5a6951aba84f5fa8f6d76a2b7fcf2d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.572037 776 raft_consensus.cc:2466] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:55.572260 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13620f6ca65342b4ad7f1ba4b437bde3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.572806 1165 raft_consensus.cc:2270] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.573311 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a31f10c8ca745b7ad635c622a265450 with delete_type TABLET_DATA_DELETED (replaced by tablet df1ed922750f45eab02d1fc0c26af75e) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.574064 1165 ts_tablet_manager.cc:1905] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:55.584450 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9bb7ebc40a14ecaa0c692a8e1117db3 consensus state has no leader
W20250625 13:51:55.585563 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af05fba6fe2042aab5dd47b30e4dfcf3 consensus state has no leader
I20250625 13:51:55.586277 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8b7062c34fd143bf86b743055a27db8c
I20250625 13:51:55.586892 594 catalog_manager.cc:6652] ReplaceTablet: tablet 8b7062c34fd143bf86b743055a27db8c deleted and replaced by tablet 6dd84b8afa2548fe8ce5925f0bf806f0
I20250625 13:51:55.587102 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b7062c34fd143bf86b743055a27db8c with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd84b8afa2548fe8ce5925f0bf806f0) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.592998 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b7062c34fd143bf86b743055a27db8c with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd84b8afa2548fe8ce5925f0bf806f0) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.593303 1174 raft_consensus.cc:491] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.593513 1174 raft_consensus.cc:513] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.594164 1174 leader_election.cc:290] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.595265 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99090fee41d44946b6243b1b4bff3635" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.595620 777 raft_consensus.cc:2466] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:55.596308 985 leader_election.cc:304] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:55.596813 1174 raft_consensus.cc:2802] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.596936 1174 raft_consensus.cc:491] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.597059 1174 raft_consensus.cc:3058] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.601321 1174 raft_consensus.cc:513] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.610394 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99090fee41d44946b6243b1b4bff3635" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:55.610688 777 raft_consensus.cc:3058] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.613103 1165 ts_tablet_manager.cc:1918] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 13:51:55.613906 1165 log.cc:1199] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/76f074156559406c875e65796e6b5ee6
I20250625 13:51:55.614131 1166 tablet_bootstrap.cc:492] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.614267 1174 leader_election.cc:290] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.614346 777 raft_consensus.cc:2466] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:55.614652 1166 ts_tablet_manager.cc:1397] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.013s
I20250625 13:51:55.571691 1307 raft_consensus.cc:3058] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.614874 985 leader_election.cc:304] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:55.615233 1174 raft_consensus.cc:2802] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.615593 1174 raft_consensus.cc:695] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:55.615528 1166 raft_consensus.cc:357] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.615744 1166 raft_consensus.cc:383] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.615818 1166 raft_consensus.cc:738] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.616082 1166 consensus_queue.cc:260] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.616451 1174 consensus_queue.cc:237] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.616554 1166 ts_tablet_manager.cc:1428] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:55.616897 1166 tablet_bootstrap.cc:492] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.572412 851 leader_election.cc:304] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
W20250625 13:51:55.617383 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 44962.1us
I20250625 13:51:55.617458 1307 raft_consensus.cc:513] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.571290 1171 raft_consensus.cc:383] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.617929 1171 raft_consensus.cc:738] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.618526 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0f2fa5d5614ad394e6e12ec62814d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:55.619560 1171 consensus_queue.cc:260] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.620805 1167 ts_tablet_manager.cc:1918] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.621208 1167 log.cc:1199] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/e75b5fb51373451a94cb5ca59cdcefac
I20250625 13:51:55.621908 1167 ts_tablet_manager.cc:1939] T e75b5fb51373451a94cb5ca59cdcefac P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:51:55.622241 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b0037d09796455cbaaadfaa9c010673 has no consensus state
I20250625 13:51:55.584254 1168 ts_tablet_manager.cc:1918] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.622782 1168 log.cc:1199] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/da4f9a46a5804da3aff1c9a87b2e4984
W20250625 13:51:55.622896 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e529f5bf13f4ee69e66402b1ee252d4 has no consensus state
I20250625 13:51:55.623196 1168 ts_tablet_manager.cc:1939] T da4f9a46a5804da3aff1c9a87b2e4984 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.624013 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 129) took 1600 ms. Trace:
I20250625 13:51:55.624138 1168 rpcz_store.cc:276] 0625 13:51:54.023372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.023405 (+ 33us) service_pool.cc:225] Handling call
0625 13:51:55.535541 (+1512136us) ts_tablet_manager.cc:1145] Deleting tablet da4f9a46a5804da3aff1c9a87b2e4984
0625 13:51:55.535556 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.566173 (+ 30617us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.584108 (+ 17935us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.623975 (+ 39867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511970}
I20250625 13:51:55.624477 1168 tablet_replica.cc:331] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.624701 1168 raft_consensus.cc:2241] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:55.624987 1168 pending_rounds.cc:70] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:51:55.624963 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_581fcddbe9bb428d84556e6156637b4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:55.625077 1168 pending_rounds.cc:77] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520571781042176 op_type: NO_OP noop_request { }
I20250625 13:51:55.625180 1168 raft_consensus.cc:2887] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:55.625291 1168 raft_consensus.cc:2270] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.625519 1367 raft_consensus.cc:2802] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.625612 1367 raft_consensus.cc:491] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.625715 1367 raft_consensus.cc:3058] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:55.626102 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_581fcddbe9bb428d84556e6156637b4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.627102 1170 tablet_bootstrap.cc:492] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.627816 1170 ts_tablet_manager.cc:1397] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20250625 13:51:55.627830 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bc6c1303fba14b97b873dd23a423bf8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.628276 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc6c1303fba14b97b873dd23a423bf8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.628484 1367 raft_consensus.cc:513] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.628806 1170 raft_consensus.cc:357] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.628976 1170 raft_consensus.cc:383] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.629061 1170 raft_consensus.cc:738] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.629118 1367 leader_election.cc:290] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.587985 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b7062c34fd143bf86b743055a27db8c with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd84b8afa2548fe8ce5925f0bf806f0) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.631376 1170 consensus_queue.cc:260] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.631544 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cd4b8059082844b99439152f1be88d22" } modify_external_catalogs: true
I20250625 13:51:55.631640 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet da4f9a46a5804da3aff1c9a87b2e4984 (table default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]) successfully deleted
I20250625 13:51:55.631738 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cd4b8059082844b99439152f1be88d22" } modify_external_catalogs: true
W20250625 13:51:55.618001 851 leader_election.cc:343] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.632017 851 leader_election.cc:304] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:51:55.632850 983 leader_election.cc:343] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 99090fee41d44946b6243b1b4bff3635
W20250625 13:51:55.633162 983 leader_election.cc:343] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 99090fee41d44946b6243b1b4bff3635
I20250625 13:51:55.633402 1170 ts_tablet_manager.cc:1428] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 13:51:55.633677 1170 tablet_bootstrap.cc:492] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.635149 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet e75b5fb51373451a94cb5ca59cdcefac (table default.Table_e75519f3b879406ea7bb8f515825f274 [id=ffb244ee24124e8a83db14b5747abed7]) successfully deleted
I20250625 13:51:55.635464 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ad21d2aa5ffb4443bb2d0b96596545c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.635571 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 04479dfb34584f0891c604ef72bf93b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04479dfb34584f0891c604ef72bf93b5 already in progress: creating tablet
I20250625 13:51:55.636691 1424 raft_consensus.cc:2747] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:55.637099 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 07e281786e6e4a5d93b94b85b486655d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07e281786e6e4a5d93b94b85b486655d already in progress: creating tablet
I20250625 13:51:55.637260 1168 ts_tablet_manager.cc:1905] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:55.637327 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6771d8e180fe48569a0e016db0cc9b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6771d8e180fe48569a0e016db0cc9b51 already in progress: creating tablet
I20250625 13:51:55.637367 1171 ts_tablet_manager.cc:1428] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.067s user 0.001s sys 0.005s
W20250625 13:51:55.637765 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 85c72f9ae92a4537979fb479eb4467c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85c72f9ae92a4537979fb479eb4467c4 already in progress: creating tablet
I20250625 13:51:55.637923 1170 tablet_bootstrap.cc:654] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.638089 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2b3cf346c6724405a0e96eef702fc43e" } modify_external_catalogs: true
I20250625 13:51:55.638703 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2b3cf346c6724405a0e96eef702fc43e" } modify_external_catalogs: true
I20250625 13:51:55.639190 1167 tablet_replica.cc:331] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:51:55.639266 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad21d2aa5ffb4443bb2d0b96596545c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.640363 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4256942aa0684fa3be579ab3ceb83fcb" } new_table_name: "default.Table_635bd0549cf242ff8169101c0efa90b6" modify_external_catalogs: true
I20250625 13:51:55.640878 1167 raft_consensus.cc:2241] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.641335 1166 tablet_bootstrap.cc:654] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.641336 1167 raft_consensus.cc:2270] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.642625 1167 ts_tablet_manager.cc:1905] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.618127 1307 leader_election.cc:290] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:51:55.618999 849 leader_election.cc:343] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.644940 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5a6951aba84f5fa8f6d76a2b7fcf2d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:55.644291 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e0f2fa5d5614ad394e6e12ec62814d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:55.645210 777 raft_consensus.cc:3058] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:55.645817 849 leader_election.cc:343] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.646724 1171 tablet_bootstrap.cc:492] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.648531 1171 tablet_bootstrap.cc:654] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.650173 777 raft_consensus.cc:2466] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:55.650588 851 leader_election.cc:304] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.652614 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c8b06a7351fb437aa0440f6fa3898939" } modify_external_catalogs: true
I20250625 13:51:55.653301 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c8b06a7351fb437aa0440f6fa3898939" } modify_external_catalogs: true
I20250625 13:51:55.654014 1307 raft_consensus.cc:2802] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.654177 1307 raft_consensus.cc:695] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:55.654430 1307 consensus_queue.cc:237] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:55.655184 593 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]). Expected version 1 got 0
I20250625 13:51:55.656103 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 2bbd509e3dfb45d98314c6afe37677f6 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.657028 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d0cf3d979369419882a2ba2083933d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.657477 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 3 got 0
I20250625 13:51:55.659091 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 637847cac1724bfeb629c4d4013396ca from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.660547 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.660972 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 3 got 0
I20250625 13:51:55.662876 1168 ts_tablet_manager.cc:1918] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:55.665051 1166 tablet_bootstrap.cc:492] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:55.665244 1166 ts_tablet_manager.cc:1397] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.001s
I20250625 13:51:55.666092 1166 raft_consensus.cc:357] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.666324 1166 raft_consensus.cc:383] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.666437 1166 raft_consensus.cc:738] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.666723 1166 consensus_queue.cc:260] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.669018 1170 tablet_bootstrap.cc:492] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.669211 1170 ts_tablet_manager.cc:1397] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.003s
I20250625 13:51:55.669973 1170 raft_consensus.cc:357] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.670157 1170 raft_consensus.cc:383] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.670233 1170 raft_consensus.cc:738] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.670497 1170 consensus_queue.cc:260] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.672168 1170 ts_tablet_manager.cc:1428] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:51:55.672447 1170 tablet_bootstrap.cc:492] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.674190 1170 tablet_bootstrap.cc:654] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.675076 1167 ts_tablet_manager.cc:1918] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.675215 1167 log.cc:1199] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/b11b3157dd764f229831c3b8417460c6
I20250625 13:51:55.675644 1167 ts_tablet_manager.cc:1939] T b11b3157dd764f229831c3b8417460c6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.676330 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 921163dc5f454c9a97e72b005f9dba19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.676668 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 99090fee41d44946b6243b1b4bff3635 with delete_type TABLET_DATA_DELETED (replaced by tablet d524089e6ab044ec961cfc9f587e8f86) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.677335 1167 tablet_replica.cc:331] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:55.677336 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 32221a413957450fa59ffcfbc91c51b9 with delete_type TABLET_DATA_DELETED (replaced by tablet bb1c7a85da844d34be805843bc60d350) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.677553 1167 raft_consensus.cc:2241] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.677735 1167 raft_consensus.cc:2270] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.677748 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet b11b3157dd764f229831c3b8417460c6 (table default.Table_8fc2c990732d4e658580c2bc2cf67f4d [id=75b34bf006e040d18266d0c90986afeb]) successfully deleted
W20250625 13:51:55.678035 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f00e72e103f443e886f0df690cff4b6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f00e72e103f443e886f0df690cff4b6f already in progress: creating tablet
I20250625 13:51:55.678483 1167 ts_tablet_manager.cc:1905] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.682940 747 tablet_service.cc:1468] Processing CreateTablet for tablet 0e77736191544e8da32ca9927f0a5804 (DEFAULT_TABLE table=default.Table_62ab081749614851902de10929497600 [id=7467e5b0513d42ff8c886b6080a5b539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.683405 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e77736191544e8da32ca9927f0a5804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.618885 1165 ts_tablet_manager.cc:1939] T 76f074156559406c875e65796e6b5ee6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:55.621618 1373 raft_consensus.cc:491] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.684769 1373 raft_consensus.cc:513] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.685549 1373 leader_election.cc:290] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.685851 1166 ts_tablet_manager.cc:1428] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
W20250625 13:51:55.686852 711 leader_election.cc:343] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: b11b3157dd764f229831c3b8417460c6
W20250625 13:51:55.687100 711 leader_election.cc:343] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: b11b3157dd764f229831c3b8417460c6
I20250625 13:51:55.687227 711 leader_election.cc:304] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:55.687665 1373 raft_consensus.cc:2747] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:55.634300 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5a6951aba84f5fa8f6d76a2b7fcf2d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:55.690903 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8fdca99b734448118e0c42227c78e58f
I20250625 13:51:55.693357 597 catalog_manager.cc:5582] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:51:55.693819 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 3 got 0
I20250625 13:51:55.694031 1167 ts_tablet_manager.cc:1918] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.694156 1167 log.cc:1199] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ce84f74470ad428abb2113efa170ca24
I20250625 13:51:55.694150 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]). Expected version 1 got 0
I20250625 13:51:55.694481 1167 ts_tablet_manager.cc:1939] T ce84f74470ad428abb2113efa170ca24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.695067 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2bbd509e3dfb45d98314c6afe37677f6
I20250625 13:51:55.695536 593 catalog_manager.cc:6652] ReplaceTablet: tablet 2bbd509e3dfb45d98314c6afe37677f6 deleted and replaced by tablet 4bd51cad7d6342d7a1d45f59653fa32f
I20250625 13:51:55.696949 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 54fa042947da4e1b89018640b46b271a from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:55.697892 1222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:51:55.698477 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8e29e7b55570483b80c6e7a1f227fa7b
I20250625 13:51:55.698566 1167 tablet_replica.cc:331] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:55.698762 1167 raft_consensus.cc:2241] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.698926 1167 raft_consensus.cc:2270] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.667928 1171 tablet_bootstrap.cc:492] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.699221 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_40ca12b3cc9a4632912f3508933960e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:55.699529 1171 ts_tablet_manager.cc:1397] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250625 13:51:55.699575 1222 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20250625 13:51:55.699939 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40ca12b3cc9a4632912f3508933960e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.700001 1222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:51:55.700398 1171 raft_consensus.cc:357] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.700578 1171 raft_consensus.cc:383] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.700664 1171 raft_consensus.cc:738] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.701001 1171 consensus_queue.cc:260] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.701599 1171 ts_tablet_manager.cc:1428] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:55.701960 1171 tablet_bootstrap.cc:492] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.702066 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_759e39cb7c154d6281c46789f97f0691" } modify_external_catalogs: true
I20250625 13:51:55.702235 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_759e39cb7c154d6281c46789f97f0691" } modify_external_catalogs: true
W20250625 13:51:55.703892 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9267df231f354c578c38f452418e523c consensus state has no leader
I20250625 13:51:55.704694 1171 tablet_bootstrap.cc:654] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.706935 895 tablet_service.cc:1468] Processing CreateTablet for tablet 0e77736191544e8da32ca9927f0a5804 (DEFAULT_TABLE table=default.Table_62ab081749614851902de10929497600 [id=7467e5b0513d42ff8c886b6080a5b539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.707147 1168 log.cc:1199] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/bf54376db991424b8de0e8f252fcb43f
I20250625 13:51:55.707412 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e77736191544e8da32ca9927f0a5804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.709121 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 1effed0fa10d409abcb31dbba54861d6 (DEFAULT_TABLE table=default.Table_ec42501f607c41ceb9d35f3dd2a38f34 [id=d2c6242f48434d3bb6c2dc01fa095a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.709553 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1effed0fa10d409abcb31dbba54861d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.709764 1167 ts_tablet_manager.cc:1905] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.711580 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ce84f74470ad428abb2113efa170ca24 (table default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]) successfully deleted
W20250625 13:51:55.711843 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c666ff2d6b9b4ed4a18e370826c143fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c666ff2d6b9b4ed4a18e370826c143fc already in progress: creating tablet
W20250625 13:51:55.712037 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7fe053e294fe4aceb24919e9e2bddf65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fe053e294fe4aceb24919e9e2bddf65 already in progress: creating tablet
W20250625 13:51:55.712208 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9a31f10c8ca745b7ad635c622a265450 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a31f10c8ca745b7ad635c622a265450 already in progress: creating tablet
I20250625 13:51:55.712252 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1effed0fa10d409abcb31dbba54861d6 (DEFAULT_TABLE table=default.Table_ec42501f607c41ceb9d35f3dd2a38f34 [id=d2c6242f48434d3bb6c2dc01fa095a67]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:55.712359 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8b7062c34fd143bf86b743055a27db8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b7062c34fd143bf86b743055a27db8c already in progress: creating tablet
I20250625 13:51:55.713932 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdca99b734448118e0c42227c78e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.714442 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (DEFAULT_TABLE table=default.Table_fa5420c9509b49d5a11c3b0a059f9f9c [id=9e881be4b5c54f5191713d2fb732d716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.714717 1027 tablet_service.cc:1468] Processing CreateTablet for tablet dc75a0a8421b457089cc707d9eb6b76f (DEFAULT_TABLE table=default.Table_ae448e34759d455882fac1dfe9789add [id=fa0246aff3f648538e6fddcb30f3a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.714861 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e529f5bf13f4ee69e66402b1ee252d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.715222 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc75a0a8421b457089cc707d9eb6b76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.715363 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd51cad7d6342d7a1d45f59653fa32f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.715657 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.715865 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.716233 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3b0037d09796455cbaaadfaa9c010673 (DEFAULT_TABLE table=default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae [id=6f1fa6d1db4e4abaae4d06299152224f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.716598 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0037d09796455cbaaadfaa9c010673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.718703 892 tablet_service.cc:1468] Processing CreateTablet for tablet 3b0037d09796455cbaaadfaa9c010673 (DEFAULT_TABLE table=default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae [id=6f1fa6d1db4e4abaae4d06299152224f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.718775 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdca99b734448118e0c42227c78e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.718838 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_683f20e141ca4122bee28d4e86aece20" } modify_external_catalogs: true
I20250625 13:51:55.719040 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_683f20e141ca4122bee28d4e86aece20" } modify_external_catalogs: true
I20250625 13:51:55.719197 896 tablet_service.cc:1468] Processing CreateTablet for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (DEFAULT_TABLE table=default.Table_fa5420c9509b49d5a11c3b0a059f9f9c [id=9e881be4b5c54f5191713d2fb732d716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.719739 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd51cad7d6342d7a1d45f59653fa32f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.719687 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8319fc497f694699988a5dfd8de276e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.720213 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8319fc497f694699988a5dfd8de276e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.720173 886 tablet_service.cc:1468] Processing CreateTablet for tablet 36b151e200e1430c99ac194bc55881c1 (DEFAULT_TABLE table=default.Table_29924a11586f4d2bb3d0f8574a7849f8 [id=e057b2ec6f914a01b474481bc5c25349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.720970 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9267df231f354c578c38f452418e523c (DEFAULT_TABLE table=default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.727018 1373 raft_consensus.cc:491] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.727177 776 raft_consensus.cc:3058] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.727813 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]) successfully deleted
I20250625 13:51:55.727214 1373 raft_consensus.cc:513] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.728259 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 637847cac1724bfeb629c4d4013396ca
I20250625 13:51:55.728724 591 catalog_manager.cc:6652] ReplaceTablet: tablet 637847cac1724bfeb629c4d4013396ca deleted and replaced by tablet b14d00c93a964edd815122e3723f89e7
I20250625 13:51:55.729463 758 tablet_service.cc:1468] Processing CreateTablet for tablet 1effed0fa10d409abcb31dbba54861d6 (DEFAULT_TABLE table=default.Table_ec42501f607c41ceb9d35f3dd2a38f34 [id=d2c6242f48434d3bb6c2dc01fa095a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.729876 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1effed0fa10d409abcb31dbba54861d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.730034 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c0a3ea18bab944749d45da9b3139f8bc
I20250625 13:51:55.732381 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09a8e15d57f14702ba0b5b52a94fc7c4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.732564 1166 tablet_bootstrap.cc:492] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:55.733273 1165 tablet_replica.cc:331] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:55.734633 887 tablet_service.cc:1515] Processing DeleteTablet for tablet cbaa018753d8455199f3869323fb5582 with delete_type TABLET_DATA_DELETED (replaced by tablet cd4f0857f52449e1b26fde09c6c7c9d4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.736294 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_20f874f96e2d4a3089c21ff64313c62a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.737515 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20f874f96e2d4a3089c21ff64313c62a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.737856 1368 raft_consensus.cc:491] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.738495 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet d4f19ddc51324cc18242c6081f3f7de5 (table default.Table_f2cf2aab62814c7389fa4b446b696dd1 [id=b8285b64bbee49f9be8741b252d03363]). Expected version 1 got 0
I20250625 13:51:55.739204 1373 leader_election.cc:290] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:55.739568 750 tablet_service.cc:1468] Processing CreateTablet for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (DEFAULT_TABLE table=default.Table_fa5420c9509b49d5a11c3b0a059f9f9c [id=9e881be4b5c54f5191713d2fb732d716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.740332 1165 raft_consensus.cc:2241] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.740535 1165 raft_consensus.cc:2270] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.740762 1368 raft_consensus.cc:513] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.741079 749 tablet_service.cc:1468] Processing CreateTablet for tablet dc75a0a8421b457089cc707d9eb6b76f (DEFAULT_TABLE table=default.Table_ae448e34759d455882fac1dfe9789add [id=fa0246aff3f648538e6fddcb30f3a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.742226 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdca99b734448118e0c42227c78e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.743361 1165 ts_tablet_manager.cc:1905] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.743942 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 54fa042947da4e1b89018640b46b271a
I20250625 13:51:55.744426 593 catalog_manager.cc:6652] ReplaceTablet: tablet 54fa042947da4e1b89018640b46b271a deleted and replaced by tablet ab7202a8761444a0a731993d890fcdeb
I20250625 13:51:55.745198 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e529f5bf13f4ee69e66402b1ee252d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.746933 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_020b99c8c2cb44e3982eaf49820a16d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.747654 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_020b99c8c2cb44e3982eaf49820a16d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.749754 752 tablet_service.cc:1468] Processing CreateTablet for tablet 3b0037d09796455cbaaadfaa9c010673 (DEFAULT_TABLE table=default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae [id=6f1fa6d1db4e4abaae4d06299152224f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.750459 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8f37509832f440ef94e577a34a689bfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.751781 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f37509832f440ef94e577a34a689bfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.752812 1368 leader_election.cc:290] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:55.753007 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd51cad7d6342d7a1d45f59653fa32f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.754041 753 tablet_service.cc:1468] Processing CreateTablet for tablet 455884b62ee44b6da53543093f721027 (DEFAULT_TABLE table=default.Table_d2296c79c2f949fca8a57149e361225e [id=67c521d398af4fc4bfe9e6c23d400906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.754443 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.755215 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc75a0a8421b457089cc707d9eb6b76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.757155 756 tablet_service.cc:1468] Processing CreateTablet for tablet 9267df231f354c578c38f452418e523c (DEFAULT_TABLE table=default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.758960 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0037d09796455cbaaadfaa9c010673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.760944 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 36b151e200e1430c99ac194bc55881c1 (DEFAULT_TABLE table=default.Table_29924a11586f4d2bb3d0f8574a7849f8 [id=e057b2ec6f914a01b474481bc5c25349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.761387 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b151e200e1430c99ac194bc55881c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.761989 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 58aa3c21191e4bea9c5395d6e32b8717
I20250625 13:51:55.763262 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.764243 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 0e77736191544e8da32ca9927f0a5804 (DEFAULT_TABLE table=default.Table_62ab081749614851902de10929497600 [id=7467e5b0513d42ff8c886b6080a5b539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.764647 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e77736191544e8da32ca9927f0a5804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.766129 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 9267df231f354c578c38f452418e523c (DEFAULT_TABLE table=default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.766443 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 8fdca99b734448118e0c42227c78e58f (DEFAULT_TABLE table=default.Table_cd4b8059082844b99439152f1be88d22 [id=d5037bf1fc844a148103da5d2ae95193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.767765 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09a8e15d57f14702ba0b5b52a94fc7c4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.770347 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8fdca99b734448118e0c42227c78e58f (DEFAULT_TABLE table=default.Table_cd4b8059082844b99439152f1be88d22 [id=d5037bf1fc844a148103da5d2ae95193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.770848 751 tablet_service.cc:1468] Processing CreateTablet for tablet 36b151e200e1430c99ac194bc55881c1 (DEFAULT_TABLE table=default.Table_29924a11586f4d2bb3d0f8574a7849f8 [id=e057b2ec6f914a01b474481bc5c25349]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:55.771939 711 leader_election.cc:343] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.772182 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 455884b62ee44b6da53543093f721027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.774883 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 637847cac1724bfeb629c4d4013396ca with delete_type TABLET_DATA_DELETED (replaced by tablet b14d00c93a964edd815122e3723f89e7) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.775144 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdca99b734448118e0c42227c78e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.777276 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet c0a3ea18bab944749d45da9b3139f8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.777805 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a7b72c2c464421eacfcf64008c3aea8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.779098 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 637847cac1724bfeb629c4d4013396ca with delete_type TABLET_DATA_DELETED (replaced by tablet b14d00c93a964edd815122e3723f89e7) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.779398 589 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 4bd51cad7d6342d7a1d45f59653fa32f
I20250625 13:51:55.779944 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 455884b62ee44b6da53543093f721027 (DEFAULT_TABLE table=default.Table_d2296c79c2f949fca8a57149e361225e [id=67c521d398af4fc4bfe9e6c23d400906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.780961 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 54fa042947da4e1b89018640b46b271a with delete_type TABLET_DATA_DELETED (replaced by tablet ab7202a8761444a0a731993d890fcdeb) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.781490 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.786403 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9267df231f354c578c38f452418e523c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.788519 1170 tablet_bootstrap.cc:492] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:55.788686 1170 ts_tablet_manager.cc:1397] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.011s
I20250625 13:51:55.790494 1170 raft_consensus.cc:357] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.790691 1170 raft_consensus.cc:383] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.791666 1170 raft_consensus.cc:738] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.791949 1170 consensus_queue.cc:260] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.793390 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 455884b62ee44b6da53543093f721027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.795128 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ac92f8b04a83459998241d6fd82297e9" } new_table_name: "default.Table_065fd6832b3b458cbedf3713e9d234b3" modify_external_catalogs: true
I20250625 13:51:55.796844 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_59ed24a1a856411ba6d9c5504c4c9a7a" } new_table_name: "default.Table_dbf3eb88d817489eae63201ef32a702c" modify_external_catalogs: true
I20250625 13:51:55.796980 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b151e200e1430c99ac194bc55881c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.798234 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f0caf6cb5cc84df9b75703d4a12eb505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.798949 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0caf6cb5cc84df9b75703d4a12eb505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.803691 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9267df231f354c578c38f452418e523c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.803718 1166 tablet_bootstrap.cc:654] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:55.808152 711 leader_election.cc:343] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:55.809695 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdca99b734448118e0c42227c78e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.812505 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 54fa042947da4e1b89018640b46b271a with delete_type TABLET_DATA_DELETED (replaced by tablet ab7202a8761444a0a731993d890fcdeb) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.812772 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.813586 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.813766 754 tablet_service.cc:1515] Processing DeleteTablet for tablet c0a3ea18bab944749d45da9b3139f8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:55.827412 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 0b129117db064a15b27e82439637cf22 (DEFAULT_TABLE table=default.Table_ad21d2aa5ffb4443bb2d0b96596545c3 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.828307 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 (DEFAULT_TABLE table=default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328 [id=1e5117fcc9bb41c3bb98021f499b766c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.828724 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 (DEFAULT_TABLE table=default.Table_bdcf17e5612a44579ba27af546ca1def [id=cc778fd879634dd6a2ba7ba8274c9ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.828806 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 09a8e15d57f14702ba0b5b52a94fc7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:55.829535 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 (DEFAULT_TABLE table=default.Table_d5d3601056d240ef8a91362211b21f7e [id=253f3f5c5b8d4051800dc823b4ebaec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.830417 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1310c8c1f7634350b0a714f5150ef5a8" } new_table_name: "default.Table_8172b8ab4c3d4d8ca80409212c6866ff" modify_external_catalogs: true
I20250625 13:51:55.832068 1167 ts_tablet_manager.cc:1918] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:55.832281 1167 log.cc:1199] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d0cf3d979369419882a2ba2083933d48
I20250625 13:51:55.832885 1167 ts_tablet_manager.cc:1939] T d0cf3d979369419882a2ba2083933d48 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:55.834102 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet d0cf3d979369419882a2ba2083933d48 (table default.Table_ac5c66505c43484ca1990febfc2a5bf9 [id=eb254e00e07047a09c1924a6cab20e4d]) successfully deleted
I20250625 13:51:55.834287 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fa5420c9509b49d5a11c3b0a059f9f9c" } new_table_name: "default.Table_934cbfc5dfe44861a26a970575a6facb" modify_external_catalogs: true
I20250625 13:51:55.836831 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 86af0ea6648045a1ac4ba903d20849b7 (DEFAULT_TABLE table=default.Table_254fead6128d4cb6b544757eb751f82b [id=65ec4be481574b20a64c98a75c6c4367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.837307 895 tablet_service.cc:1468] Processing CreateTablet for tablet 86af0ea6648045a1ac4ba903d20849b7 (DEFAULT_TABLE table=default.Table_254fead6128d4cb6b544757eb751f82b [id=65ec4be481574b20a64c98a75c6c4367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.837447 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 04dd85668a47461e9e82f8b251468ad3 (DEFAULT_TABLE table=default.Table_144bca3dd4db415b876a78a3b468e320 [id=687938ac1bb04d52a46b554e357b35ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.839052 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 (DEFAULT_TABLE table=default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.840634 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 063a793e0d7747668d2728fdc2160b5e (DEFAULT_TABLE table=default.Table_f41a4c203a604e7786bb8716dde80fd3 [id=3398f6ade26340dcae5b0da2eb58034c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.841442 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b129117db064a15b27e82439637cf22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.842190 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd84b8afa2548fe8ce5925f0bf806f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.842573 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d43ba17ebfec4244894d1b6e23200c33 (DEFAULT_TABLE table=default.Table_bc6c1303fba14b97b873dd23a423bf8c [id=999e931fc6724eec9b6d8bb94bd3cf23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.842949 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43ba17ebfec4244894d1b6e23200c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.844055 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063a793e0d7747668d2728fdc2160b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.849987 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (DEFAULT_TABLE table=default.Table_f728647d61a4414ca11ff2b890235012 [id=71729b0c8f7046bb8e80bb96f9f2ba71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.850986 1167 tablet_replica.cc:331] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:51:55.852131 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1acc41a93f6433588727811903921fe consensus state has no leader
I20250625 13:51:55.852280 1167 raft_consensus.cc:2241] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.852916 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1effed0fa10d409abcb31dbba54861d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.853212 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b151e200e1430c99ac194bc55881c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.853874 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.854305 885 tablet_service.cc:1468] Processing CreateTablet for tablet 04dd85668a47461e9e82f8b251468ad3 (DEFAULT_TABLE table=default.Table_144bca3dd4db415b876a78a3b468e320 [id=687938ac1bb04d52a46b554e357b35ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.854363 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0892d2b459334a8bac680a69f2be033e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:55.854737 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd85668a47461e9e82f8b251468ad3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:55.854903 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0892d2b459334a8bac680a69f2be033e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.855383 890 tablet_service.cc:1468] Processing CreateTablet for tablet dc75a0a8421b457089cc707d9eb6b76f (DEFAULT_TABLE table=default.Table_ae448e34759d455882fac1dfe9789add [id=fa0246aff3f648538e6fddcb30f3a399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.855760 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc75a0a8421b457089cc707d9eb6b76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.859373 883 tablet_service.cc:1468] Processing CreateTablet for tablet 455884b62ee44b6da53543093f721027 (DEFAULT_TABLE table=default.Table_d2296c79c2f949fca8a57149e361225e [id=67c521d398af4fc4bfe9e6c23d400906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.859728 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 455884b62ee44b6da53543093f721027. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.863521 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9267df231f354c578c38f452418e523c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.865592 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b0037d09796455cbaaadfaa9c010673. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.869472 888 tablet_service.cc:1468] Processing CreateTablet for tablet 8fdca99b734448118e0c42227c78e58f (DEFAULT_TABLE table=default.Table_cd4b8059082844b99439152f1be88d22 [id=d5037bf1fc844a148103da5d2ae95193]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.869616 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 637847cac1724bfeb629c4d4013396ca with delete_type TABLET_DATA_DELETED (replaced by tablet b14d00c93a964edd815122e3723f89e7) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.869725 915 raft_consensus.cc:2466] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:55.870038 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdca99b734448118e0c42227c78e58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.870092 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 (DEFAULT_TABLE table=default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.870313 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a7b72c2c464421eacfcf64008c3aea8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.870442 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd84b8afa2548fe8ce5925f0bf806f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.870527 916 raft_consensus.cc:2466] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:55.869570 894 tablet_service.cc:1515] Processing DeleteTablet for tablet c0a3ea18bab944749d45da9b3139f8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.872831 894 tablet_service.cc:1468] Processing CreateTablet for tablet 063a793e0d7747668d2728fdc2160b5e (DEFAULT_TABLE table=default.Table_f41a4c203a604e7786bb8716dde80fd3 [id=3398f6ade26340dcae5b0da2eb58034c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.871742 1174 raft_consensus.cc:491] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.875483 1174 raft_consensus.cc:513] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.876073 1168 ts_tablet_manager.cc:1939] T bf54376db991424b8de0e8f252fcb43f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:55.876276 1174 leader_election.cc:290] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.872560 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 067272869c5547648c90f83a34a036aa (DEFAULT_TABLE table=default.Table_10094a18787042588bd628e824da998f [id=a1f1874f2c0d454694a40c5d5fcf4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.877554 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e529f5bf13f4ee69e66402b1ee252d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.878055 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32221a413957450fa59ffcfbc91c51b9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:55.878911 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 055f88e118454e32baa578cae9aceb0c consensus state has no leader
I20250625 13:51:55.878957 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86af0ea6648045a1ac4ba903d20849b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.880266 1171 tablet_bootstrap.cc:492] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:55.872589 1170 ts_tablet_manager.cc:1428] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20250625 13:51:55.880709 893 tablet_service.cc:1468] Processing CreateTablet for tablet d43ba17ebfec4244894d1b6e23200c33 (DEFAULT_TABLE table=default.Table_bc6c1303fba14b97b873dd23a423bf8c [id=999e931fc6724eec9b6d8bb94bd3cf23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.881125 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43ba17ebfec4244894d1b6e23200c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.881482 776 raft_consensus.cc:2466] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:55.852777 1167 raft_consensus.cc:2270] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.886270 711 leader_election.cc:304] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.886803 711 leader_election.cc:304] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
W20250625 13:51:55.887965 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e529f5bf13f4ee69e66402b1ee252d4 consensus state has no leader
I20250625 13:51:55.888898 1368 raft_consensus.cc:491] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.890118 1167 ts_tablet_manager.cc:1905] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:55.890372 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cf73b4ab99c466abcfedf1b200e743b consensus state has no leader
I20250625 13:51:55.890504 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063a793e0d7747668d2728fdc2160b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.890643 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd85668a47461e9e82f8b251468ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.892822 886 tablet_service.cc:1468] Processing CreateTablet for tablet dad4ea8cf44944b08f53428c95809daf (DEFAULT_TABLE table=default.Table_4a201c5623af4b8a95f24f43319311f0 [id=a249a45df8b3454894e648d70f5e8b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.895328 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c50e1ebb9042ef8f0a651ba7186b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.895587 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa801f7d98e4794b6ca439638f29e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.898249 1373 raft_consensus.cc:2802] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.898100 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 5cf73b4ab99c466abcfedf1b200e743b (DEFAULT_TABLE table=default.Table_1310c8c1f7634350b0a714f5150ef5a8 [id=8c0cf5af3aee43e9b7676c1be4842093]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.899480 1174 raft_consensus.cc:491] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.899806 754 tablet_service.cc:1468] Processing CreateTablet for tablet d43ba17ebfec4244894d1b6e23200c33 (DEFAULT_TABLE table=default.Table_bc6c1303fba14b97b873dd23a423bf8c [id=999e931fc6724eec9b6d8bb94bd3cf23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.900280 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43ba17ebfec4244894d1b6e23200c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.900861 1013 tablet_service.cc:1468] Processing CreateTablet for tablet dad4ea8cf44944b08f53428c95809daf (DEFAULT_TABLE table=default.Table_4a201c5623af4b8a95f24f43319311f0 [id=a249a45df8b3454894e648d70f5e8b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.901261 1174 raft_consensus.cc:513] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:55.902318 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86af0ea6648045a1ac4ba903d20849b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.903865 1368 raft_consensus.cc:513] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.904731 1368 leader_election.cc:290] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:55.905678 1302 raft_consensus.cc:2802] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.905808 1302 raft_consensus.cc:491] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.907059 1302 raft_consensus.cc:3058] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.907536 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_194007e5294849f9a6734e39b20d4fff" } modify_external_catalogs: true
I20250625 13:51:55.907727 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_194007e5294849f9a6734e39b20d4fff" } modify_external_catalogs: true
I20250625 13:51:55.908066 746 tablet_service.cc:1468] Processing CreateTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 (DEFAULT_TABLE table=default.Table_bdcf17e5612a44579ba27af546ca1def [id=cc778fd879634dd6a2ba7ba8274c9ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.908897 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c50e1ebb9042ef8f0a651ba7186b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.909099 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1be3d9db61ab4b56b98b70af09df98b3" } modify_external_catalogs: true
I20250625 13:51:55.909271 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1be3d9db61ab4b56b98b70af09df98b3" } modify_external_catalogs: true
I20250625 13:51:55.910466 884 tablet_service.cc:1468] Processing CreateTablet for tablet 5cf73b4ab99c466abcfedf1b200e743b (DEFAULT_TABLE table=default.Table_1310c8c1f7634350b0a714f5150ef5a8 [id=8c0cf5af3aee43e9b7676c1be4842093]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.910828 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0b129117db064a15b27e82439637cf22 (DEFAULT_TABLE table=default.Table_ad21d2aa5ffb4443bb2d0b96596545c3 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.911183 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b129117db064a15b27e82439637cf22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.912154 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2211bebab65640d09cfecd34b8eaaa40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.912650 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2211bebab65640d09cfecd34b8eaaa40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.914013 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f41a4c203a604e7786bb8716dde80fd3" } new_table_name: "default.Table_96a7158ed5964f8aaff3f72fc357f629" modify_external_catalogs: true
I20250625 13:51:55.892557 877 tablet_service.cc:1468] Processing CreateTablet for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (DEFAULT_TABLE table=default.Table_f728647d61a4414ca11ff2b890235012 [id=71729b0c8f7046bb8e80bb96f9f2ba71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.915704 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_25d9c73fdf1b43b787345769f54915d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.916878 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25d9c73fdf1b43b787345769f54915d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.917589 750 tablet_service.cc:1468] Processing CreateTablet for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (DEFAULT_TABLE table=default.Table_f728647d61a4414ca11ff2b890235012 [id=71729b0c8f7046bb8e80bb96f9f2ba71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.880182 882 tablet_service.cc:1468] Processing CreateTablet for tablet 0b129117db064a15b27e82439637cf22 (DEFAULT_TABLE table=default.Table_ad21d2aa5ffb4443bb2d0b96596545c3 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.918895 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fe053e294fe4aceb24919e9e2bddf65" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.920809 748 tablet_service.cc:1468] Processing CreateTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 (DEFAULT_TABLE table=default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328 [id=1e5117fcc9bb41c3bb98021f499b766c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.920846 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 (DEFAULT_TABLE table=default.Table_d5d3601056d240ef8a91362211b21f7e [id=253f3f5c5b8d4051800dc823b4ebaec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.921145 757 tablet_service.cc:1468] Processing CreateTablet for tablet 067272869c5547648c90f83a34a036aa (DEFAULT_TABLE table=default.Table_10094a18787042588bd628e824da998f [id=a1f1874f2c0d454694a40c5d5fcf4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.923260 1373 raft_consensus.cc:491] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.923344 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 54fa042947da4e1b89018640b46b271a with delete_type TABLET_DATA_DELETED (replaced by tablet ab7202a8761444a0a731993d890fcdeb) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:55.923767 891 tablet_service.cc:1468] Processing CreateTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 (DEFAULT_TABLE table=default.Table_f5662b7c3e9a430d87a09930ab6e8f5c [id=7b67603755e7428496524ee716b435e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.925771 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32221a413957450fa59ffcfbc91c51b9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.926034 916 raft_consensus.cc:2391] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:51:55.925988 1174 leader_election.cc:290] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.927904 747 tablet_service.cc:1468] Processing CreateTablet for tablet 04dd85668a47461e9e82f8b251468ad3 (DEFAULT_TABLE table=default.Table_144bca3dd4db415b876a78a3b468e320 [id=687938ac1bb04d52a46b554e357b35ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.928246 1307 raft_consensus.cc:491] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.928642 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "412a0c5f05ab489ab9f8fc3f44fa7bdc" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:55.929234 885 tablet_service.cc:1468] Processing CreateTablet for tablet a0e81d4c904248eba858462bcf31c5d9 (DEFAULT_TABLE table=default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.929526 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 130) took 1895 ms. Trace:
I20250625 13:51:55.882529 880 tablet_service.cc:1468] Processing CreateTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 (DEFAULT_TABLE table=default.Table_bdcf17e5612a44579ba27af546ca1def [id=cc778fd879634dd6a2ba7ba8274c9ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.930848 1367 raft_consensus.cc:491] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:55.931303 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_221b43b171de43b19b753208d05a4e40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:55.931877 887 tablet_service.cc:1468] Processing CreateTablet for tablet df1ed922750f45eab02d1fc0c26af75e (DEFAULT_TABLE table=default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:55.931950 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_221b43b171de43b19b753208d05a4e40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.932090 983 leader_election.cc:304] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:55.933666 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_01ec4a7ff073444b929c6c217632a150" } modify_external_catalogs: true
I20250625 13:51:55.933863 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_01ec4a7ff073444b929c6c217632a150" } modify_external_catalogs: true
I20250625 13:51:55.934607 1171 ts_tablet_manager.cc:1397] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.179s user 0.017s sys 0.032s
I20250625 13:51:55.884490 881 tablet_service.cc:1468] Processing CreateTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 (DEFAULT_TABLE table=default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328 [id=1e5117fcc9bb41c3bb98021f499b766c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.884204 879 tablet_service.cc:1468] Processing CreateTablet for tablet 067272869c5547648c90f83a34a036aa (DEFAULT_TABLE table=default.Table_10094a18787042588bd628e824da998f [id=a1f1874f2c0d454694a40c5d5fcf4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.936179 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad4ea8cf44944b08f53428c95809daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.937039 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf73b4ab99c466abcfedf1b200e743b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.937201 1373 raft_consensus.cc:3058] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.937320 1171 raft_consensus.cc:357] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.937613 1307 raft_consensus.cc:513] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.938373 1171 raft_consensus.cc:383] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:55.938498 1171 raft_consensus.cc:738] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:55.938803 878 tablet_service.cc:1468] Processing CreateTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 (DEFAULT_TABLE table=default.Table_d5d3601056d240ef8a91362211b21f7e [id=253f3f5c5b8d4051800dc823b4ebaec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.941255 758 tablet_service.cc:1468] Processing CreateTablet for tablet 86af0ea6648045a1ac4ba903d20849b7 (DEFAULT_TABLE table=default.Table_254fead6128d4cb6b544757eb751f82b [id=65ec4be481574b20a64c98a75c6c4367]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.942271 776 raft_consensus.cc:2391] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:51:55.942945 1171 consensus_queue.cc:260] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.947511 851 leader_election.cc:304] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:55.942946 1367 raft_consensus.cc:513] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.951066 1171 ts_tablet_manager.cc:1428] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20250625 13:51:55.951387 743 tablet_service.cc:1468] Processing CreateTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 (DEFAULT_TABLE table=default.Table_f5662b7c3e9a430d87a09930ab6e8f5c [id=7b67603755e7428496524ee716b435e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.951026 744 tablet_service.cc:1468] Processing CreateTablet for tablet dad4ea8cf44944b08f53428c95809daf (DEFAULT_TABLE table=default.Table_4a201c5623af4b8a95f24f43319311f0 [id=a249a45df8b3454894e648d70f5e8b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.953040 1424 raft_consensus.cc:2802] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:55.956001 1367 leader_election.cc:290] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.952395 742 tablet_service.cc:1468] Processing CreateTablet for tablet a0e81d4c904248eba858462bcf31c5d9 (DEFAULT_TABLE table=default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.958235 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f4ab201a21c4415fa8b6ac70e579492b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:55.958971 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4ab201a21c4415fa8b6ac70e579492b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:55.952811 745 tablet_service.cc:1468] Processing CreateTablet for tablet 5cf73b4ab99c466abcfedf1b200e743b (DEFAULT_TABLE table=default.Table_1310c8c1f7634350b0a714f5150ef5a8 [id=8c0cf5af3aee43e9b7676c1be4842093]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.959370 915 raft_consensus.cc:2466] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:55.960357 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f56a8a6cb964d3a979e2f1d944eeddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.960764 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be3313ec5dfb41b68662d3ff4c8c94f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.961714 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa801f7d98e4794b6ca439638f29e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.962831 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be3313ec5dfb41b68662d3ff4c8c94f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.963106 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067272869c5547648c90f83a34a036aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.963498 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f56a8a6cb964d3a979e2f1d944eeddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.963568 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067272869c5547648c90f83a34a036aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.964162 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad4ea8cf44944b08f53428c95809daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.965008 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c1deb52366c404cbdc08076d4bcb3d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.965237 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_29924a11586f4d2bb3d0f8574a7849f8" } modify_external_catalogs: true
I20250625 13:51:55.949561 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c1deb52366c404cbdc08076d4bcb3d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.965423 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_29924a11586f4d2bb3d0f8574a7849f8" } modify_external_catalogs: true
I20250625 13:51:55.966008 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd85668a47461e9e82f8b251468ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.966027 711 leader_election.cc:304] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:55.966372 1368 raft_consensus.cc:2802] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:55.966464 1368 raft_consensus.cc:491] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:55.966586 1368 raft_consensus.cc:3058] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.966761 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86af0ea6648045a1ac4ba903d20849b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.969970 741 tablet_service.cc:1468] Processing CreateTablet for tablet df1ed922750f45eab02d1fc0c26af75e (DEFAULT_TABLE table=default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.971886 1368 raft_consensus.cc:513] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:55.972563 1168 rpcz_store.cc:276] 0625 13:51:54.023686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.023729 (+ 43us) service_pool.cc:225] Handling call
0625 13:51:55.624418 (+1600689us) ts_tablet_manager.cc:1145] Deleting tablet bf54376db991424b8de0e8f252fcb43f
0625 13:51:55.624428 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.649039 (+ 24611us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.662734 (+ 13695us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:55.919485 (+256751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600503}
I20250625 13:51:55.972936 1168 tablet_replica.cc:331] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:55.973685 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet bf54376db991424b8de0e8f252fcb43f (table default.Table_719d67c948c8495ba1f5c2001c020ce5 [id=8504f82376b440a9a3ab8c9bf1741cfe]) successfully deleted
I20250625 13:51:55.974665 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e81d4c904248eba858462bcf31c5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.978629 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf73b4ab99c466abcfedf1b200e743b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.976845 1424 raft_consensus.cc:695] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:55.979372 1171 tablet_bootstrap.cc:492] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:55.979681 1307 leader_election.cc:290] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:55.980408 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c50e1ebb9042ef8f0a651ba7186b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.981043 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6199817767f74031b8a500d21174d5a6" } modify_external_catalogs: true
I20250625 13:51:55.981248 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6199817767f74031b8a500d21174d5a6" } modify_external_catalogs: true
I20250625 13:51:55.950397 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "412a0c5f05ab489ab9f8fc3f44fa7bdc" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:55.982514 1011 tablet_service.cc:1468] Processing CreateTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 (DEFAULT_TABLE table=default.Table_f5662b7c3e9a430d87a09930ab6e8f5c [id=7b67603755e7428496524ee716b435e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:55.983434 1168 raft_consensus.cc:2241] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:55.983639 1168 raft_consensus.cc:2270] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:55.984479 1168 ts_tablet_manager.cc:1905] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:55.985407 1368 leader_election.cc:290] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:55.985711 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fe053e294fe4aceb24919e9e2bddf65" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.986284 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fe053e294fe4aceb24919e9e2bddf65" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:55.986478 915 raft_consensus.cc:3058] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.989467 915 raft_consensus.cc:2466] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:55.990114 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbaa018753d8455199f3869323fb5582" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.990562 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 067272869c5547648c90f83a34a036aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.993235 711 leader_election.cc:304] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:55.993311 1170 tablet_bootstrap.cc:492] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:55.993410 1174 raft_consensus.cc:3058] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:55.994019 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be3313ec5dfb41b68662d3ff4c8c94f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.993242 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5786bb7716ef44689ef9862fc3ce17e9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:55.994258 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1ed922750f45eab02d1fc0c26af75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:55.997718 1302 raft_consensus.cc:513] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:56.001634 1368 raft_consensus.cc:2802] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:56.001678 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c1deb52366c404cbdc08076d4bcb3d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.002302 1368 raft_consensus.cc:695] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:56.002374 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad4ea8cf44944b08f53428c95809daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.003443 1368 consensus_queue.cc:237] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.004069 1170 tablet_bootstrap.cc:654] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:55.979530 1424 consensus_queue.cc:237] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:56.006778 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf73b4ab99c466abcfedf1b200e743b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.011368 1028 tablet_service.cc:1468] Processing CreateTablet for tablet a0e81d4c904248eba858462bcf31c5d9 (DEFAULT_TABLE table=default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.014690 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa801f7d98e4794b6ca439638f29e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.020051 1018 tablet_service.cc:1468] Processing CreateTablet for tablet df1ed922750f45eab02d1fc0c26af75e (DEFAULT_TABLE table=default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.027602 1367 raft_consensus.cc:491] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.027767 1367 raft_consensus.cc:513] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.028398 1367 leader_election.cc:290] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:56.029228 1307 raft_consensus.cc:491] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.030973 1307 raft_consensus.cc:513] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:56.031631 1307 leader_election.cc:290] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:56.032148 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b129117db064a15b27e82439637cf22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.033285 1373 raft_consensus.cc:513] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.033717 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e81d4c904248eba858462bcf31c5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.034027 1373 leader_election.cc:290] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.034710 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1ed922750f45eab02d1fc0c26af75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.039746 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 (DEFAULT_TABLE table=default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.040372 751 tablet_service.cc:1468] Processing CreateTablet for tablet 063a793e0d7747668d2728fdc2160b5e (DEFAULT_TABLE table=default.Table_f41a4c203a604e7786bb8716dde80fd3 [id=3398f6ade26340dcae5b0da2eb58034c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.044406 1302 leader_election.cc:290] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:56.049453 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fe053e294fe4aceb24919e9e2bddf65" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:56.049654 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:56.049880 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a7b72c2c464421eacfcf64008c3aea8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:51:56.050191 711 leader_election.cc:343] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.050246 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09a8e15d57f14702ba0b5b52a94fc7c4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:56.050482 1046 raft_consensus.cc:3058] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.054487 1046 raft_consensus.cc:2466] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:56.055866 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b549bc8cead4af4a72f41f51655602c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:56.056358 711 leader_election.cc:343] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:56.056725 711 leader_election.cc:343] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.056879 711 leader_election.cc:304] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:56.057148 1302 raft_consensus.cc:2802] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:56.059604 1302 raft_consensus.cc:695] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:56.060631 1302 consensus_queue.cc:237] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:56.055883 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a7b72c2c464421eacfcf64008c3aea8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:56.081370 1165 ts_tablet_manager.cc:1918] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.081686 1165 log.cc:1199] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/67441795e22b445a90c88d76aa519ea5
I20250625 13:51:56.081818 1174 raft_consensus.cc:2747] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:56.082484 1165 ts_tablet_manager.cc:1939] T 67441795e22b445a90c88d76aa519ea5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.083029 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09a8e15d57f14702ba0b5b52a94fc7c4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:56.083693 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 191) took 1379 ms. Trace:
I20250625 13:51:56.083873 1165 rpcz_store.cc:276] 0625 13:51:54.704539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.706029 (+ 1490us) service_pool.cc:225] Handling call
0625 13:51:55.728297 (+1022268us) ts_tablet_manager.cc:1145] Deleting tablet 67441795e22b445a90c88d76aa519ea5
0625 13:51:55.728307 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:55.956456 (+228149us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.063080 (+106624us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.083605 (+ 20525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022058}
I20250625 13:51:56.084214 915 raft_consensus.cc:3058] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:56.082582 849 leader_election.cc:343] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.084609 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 200) took 1250 ms. Trace:
I20250625 13:51:56.085247 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 67441795e22b445a90c88d76aa519ea5 (table default.Table_5a2f402bff584ef78d361f827bcd56f7 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]) successfully deleted
I20250625 13:51:56.085719 755 tablet_service.cc:1468] Processing CreateTablet for tablet ab7202a8761444a0a731993d890fcdeb (DEFAULT_TABLE table=default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.086242 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7202a8761444a0a731993d890fcdeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.087615 914 raft_consensus.cc:3058] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.087625 756 tablet_service.cc:1468] Processing CreateTablet for tablet b14d00c93a964edd815122e3723f89e7 (DEFAULT_TABLE table=default.Table_a07c15abd0224e7d80d182ea2c891993 [id=47cd45886fc94cd7ba5cafa60838d601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.087949 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b14d00c93a964edd815122e3723f89e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.090955 740 tablet_service.cc:1468] Processing CreateTablet for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (DEFAULT_TABLE table=default.Table_40ca12b3cc9a4632912f3508933960e2 [id=25a7472e96b4448683b5fcf35e540048]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.091485 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79beb23eebf54fd9951ea5a33fcfb0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.091156 739 tablet_service.cc:1468] Processing CreateTablet for tablet ac4d368347444846880954ced9651404 (DEFAULT_TABLE table=default.Table_581fcddbe9bb428d84556e6156637b4a [id=3a7ba6e0a96c4f3fb656f9ac4f6ee82d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:56.091872 849 leader_election.cc:343] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:56.092145 849 leader_election.cc:343] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.092293 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d368347444846880954ced9651404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.093403 1170 tablet_bootstrap.cc:492] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:56.093575 1170 ts_tablet_manager.cc:1397] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.001s
I20250625 13:51:56.094347 1170 raft_consensus.cc:357] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.094583 1170 raft_consensus.cc:383] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.094666 1170 raft_consensus.cc:738] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.094902 1170 consensus_queue.cc:260] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.099730 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7108932fa8a241819ca90674a64b285e
I20250625 13:51:56.101667 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_dacf434b035e41c293c9fb0057989436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.102218 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dacf434b035e41c293c9fb0057989436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.102408 752 tablet_service.cc:1468] Processing CreateTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 (DEFAULT_TABLE table=default.Table_020b99c8c2cb44e3982eaf49820a16d0 [id=5fcd0958c61942289b4eb28c077ceec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.102929 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a23b08a03b48eaa051f670eb7dac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.106269 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e81d4c904248eba858462bcf31c5d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.108146 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 063a793e0d7747668d2728fdc2160b5e consensus state has no leader
I20250625 13:51:56.110113 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3462a32d0970423c8538a79dae984471"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.110658 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3462a32d0970423c8538a79dae984471 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.111717 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_56e4bf4e04984c3db6effdb227c7a75d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.112243 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56e4bf4e04984c3db6effdb227c7a75d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.114243 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd84b8afa2548fe8ce5925f0bf806f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.116712 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 7108932fa8a241819ca90674a64b285e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.120433 745 tablet_service.cc:1468] Processing CreateTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 (DEFAULT_TABLE table=default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.120944 753 tablet_service.cc:1468] Processing CreateTablet for tablet 430c3bed360b4f32855a8adb99076608 (DEFAULT_TABLE table=default.Table_8319fc497f694699988a5dfd8de276e0 [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.121347 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063a793e0d7747668d2728fdc2160b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.127430 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f56a8a6cb964d3a979e2f1d944eeddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.128912 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 430c3bed360b4f32855a8adb99076608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.131426 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f5b36ddb474ed6903c14876b236ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.134001 1166 tablet_bootstrap.cc:492] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.134213 1166 ts_tablet_manager.cc:1397] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.443s user 0.017s sys 0.008s
W20250625 13:51:56.134316 849 leader_election.cc:343] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.084695 1165 rpcz_store.cc:276] 0625 13:51:54.834452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.837804 (+ 3352us) service_pool.cc:225] Handling call
0625 13:51:56.084422 (+1246618us) ts_tablet_manager.cc:1145] Deleting tablet d0c5bea3b6214fa8af2792c5e9c411d9
0625 13:51:56.084430 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.084552 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1246151}
I20250625 13:51:56.134718 1367 raft_consensus.cc:491] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:51:56.135025 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d0c5bea3b6214fa8af2792c5e9c411d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0c5bea3b6214fa8af2792c5e9c411d9 already in progress: creating tablet
I20250625 13:51:56.134954 1367 raft_consensus.cc:513] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.135511 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 199) took 1301 ms. Trace:
I20250625 13:51:56.135603 1165 rpcz_store.cc:276] 0625 13:51:54.834119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.855832 (+ 21713us) service_pool.cc:225] Handling call
0625 13:51:56.135349 (+1279517us) ts_tablet_manager.cc:1145] Deleting tablet 15e58fbec94e4ff9b7ac9830b576fe7f
0625 13:51:56.135360 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.135458 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278626}
I20250625 13:51:56.135751 1307 raft_consensus.cc:491] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.135954 1307 raft_consensus.cc:513] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:56.136039 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 15e58fbec94e4ff9b7ac9830b576fe7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15e58fbec94e4ff9b7ac9830b576fe7f already in progress: creating tablet
I20250625 13:51:56.136536 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 204) took 1269 ms. Trace:
I20250625 13:51:56.136617 1165 rpcz_store.cc:276] 0625 13:51:54.867049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.867103 (+ 54us) service_pool.cc:225] Handling call
0625 13:51:56.136370 (+1269267us) ts_tablet_manager.cc:1145] Deleting tablet 2cd5303e674246ada7d36574b535b333
0625 13:51:56.136381 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.136492 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268970}
W20250625 13:51:56.136987 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2cd5303e674246ada7d36574b535b333 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd5303e674246ada7d36574b535b333 already in progress: creating tablet
I20250625 13:51:56.137511 1167 ts_tablet_manager.cc:1918] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.120179 743 tablet_service.cc:1468] Processing CreateTablet for tablet 2404e712e18146b68757bd77362c1358 (DEFAULT_TABLE table=default.Table_8f37509832f440ef94e577a34a689bfe [id=0532faedda28481f9742008aecb6bfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.140383 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2404e712e18146b68757bd77362c1358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.147936 1165 tablet_replica.cc:331] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.148500 1165 raft_consensus.cc:2241] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:56.149386 1165 pending_rounds.cc:70] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:56.154667 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_de6ae2243e604e0a8094e43c5b76ca27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.155298 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de6ae2243e604e0a8094e43c5b76ca27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.161479 1367 leader_election.cc:290] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:56.162719 1167 log.cc:1199] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/921163dc5f454c9a97e72b005f9dba19
I20250625 13:51:56.163374 1165 pending_rounds.cc:77] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520575160500224 op_type: NO_OP noop_request { }
I20250625 13:51:56.163967 1165 raft_consensus.cc:2887] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:56.164371 1165 raft_consensus.cc:2270] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.165436 1165 ts_tablet_manager.cc:1905] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.170490 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ae448e34759d455882fac1dfe9789add" } new_table_name: "default.Table_7c7ffa1aa92d44d7a29c530d6b9279e6" modify_external_catalogs: true
W20250625 13:51:56.172080 711 consensus_peers.cc:487] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0cf3d979369419882a2ba2083933d48. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:56.145809 1170 ts_tablet_manager.cc:1428] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20250625 13:51:56.173645 1170 tablet_bootstrap.cc:492] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:56.139856 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67441795e22b445a90c88d76aa519ea5" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:56.175865 1167 ts_tablet_manager.cc:1939] T 921163dc5f454c9a97e72b005f9dba19 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:56.176908 1170 tablet_bootstrap.cc:654] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.181810 1307 leader_election.cc:290] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:51:56.181981 849 leader_election.cc:343] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.184580 914 raft_consensus.cc:2466] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:56.166916 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a21335a06ec04da29b3351554c75f03c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:56.200042 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 13620f6ca65342b4ad7f1ba4b437bde3
W20250625 13:51:56.200253 849 leader_election.cc:343] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.201968 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 13620f6ca65342b4ad7f1ba4b437bde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.204648 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bc823b8c0cbd4aa8a4c7465e9e36410a" } new_table_name: "default.Table_7a2ca389ecd148b9be0e90c130fe6431" modify_external_catalogs: true
I20250625 13:51:56.204778 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6b0fc9a5299d475aa3ac845d06953e17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.205379 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b0fc9a5299d475aa3ac845d06953e17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.205984 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df1ed922750f45eab02d1fc0c26af75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.212152 1166 raft_consensus.cc:357] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.212399 1166 raft_consensus.cc:383] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.212478 1166 raft_consensus.cc:738] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.213182 1166 consensus_queue.cc:260] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.213832 1166 ts_tablet_manager.cc:1428] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.079s user 0.001s sys 0.001s
I20250625 13:51:56.215939 1166 tablet_bootstrap.cc:492] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.217600 1166 tablet_bootstrap.cc:654] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.164443 1174 raft_consensus.cc:491] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.220683 1174 raft_consensus.cc:513] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:56.222128 1174 leader_election.cc:290] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:56.234359 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1f6be7189f8f45ccb329ef5d2fd8da8d
I20250625 13:51:56.235517 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 36b151e200e1430c99ac194bc55881c1
I20250625 13:51:56.236364 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bd5ba76c130a4711af5b21e9eb8562e7
I20250625 13:51:56.222975 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5a6951aba84f5fa8f6d76a2b7fcf2d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:56.239598 776 raft_consensus.cc:2391] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:51:56.240525 1171 tablet_bootstrap.cc:654] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.247454 1165 ts_tablet_manager.cc:1918] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:56.249650 1168 ts_tablet_manager.cc:1918] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:56.249788 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.250232 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 36b151e200e1430c99ac194bc55881c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.250782 753 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.253266 1165 log.cc:1199] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/32221a413957450fa59ffcfbc91c51b9
I20250625 13:51:56.254628 1165 ts_tablet_manager.cc:1939] T 32221a413957450fa59ffcfbc91c51b9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.255940 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 205) took 1379 ms. Trace:
I20250625 13:51:56.259068 591 master_service.cc:948] ReplaceTablet: received request to replace tablet d0cf3d979369419882a2ba2083933d48 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:56.259485 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 01ba67b500b646d7aa55315f7ee23ed6 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:56.259805 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_776dedc025c447b3a2709df16fa5d505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.256598 1166 tablet_bootstrap.cc:492] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
W20250625 13:51:56.260336 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_776dedc025c447b3a2709df16fa5d505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.260488 1166 ts_tablet_manager.cc:1397] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20250625 13:51:56.261679 1166 raft_consensus.cc:357] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.262475 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8f37509832f440ef94e577a34a689bfe" } new_table_name: "default.Table_9b1b976f4c234b97923637678b6508f9" modify_external_catalogs: true
I20250625 13:51:56.263077 1166 raft_consensus.cc:383] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.263185 1166 raft_consensus.cc:738] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.263432 1166 consensus_queue.cc:260] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.263716 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5a2f402bff584ef78d361f827bcd56f7" } new_table_name: "default.Table_5ca183162fa545e7bd03a6bba000e7c9" modify_external_catalogs: true
I20250625 13:51:56.264987 1166 ts_tablet_manager.cc:1428] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:56.265259 1166 tablet_bootstrap.cc:492] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.266175 1170 tablet_bootstrap.cc:492] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:56.267206 1170 ts_tablet_manager.cc:1397] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.094s user 0.007s sys 0.000s
I20250625 13:51:56.268814 1166 tablet_bootstrap.cc:654] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.269580 1165 rpcz_store.cc:276] 0625 13:51:54.876178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.876242 (+ 64us) service_pool.cc:225] Handling call
0625 13:51:56.147802 (+1271560us) ts_tablet_manager.cc:1145] Deleting tablet 32221a413957450fa59ffcfbc91c51b9
0625 13:51:56.147811 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.224232 (+ 76421us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.247250 (+ 23018us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.255886 (+ 8636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1271032}
I20250625 13:51:56.270097 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 67441795e22b445a90c88d76aa519ea5 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:56.271800 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 32221a413957450fa59ffcfbc91c51b9 (table default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]) successfully deleted
I20250625 13:51:56.272936 1170 raft_consensus.cc:357] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.273312 1368 raft_consensus.cc:491] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.273450 1368 raft_consensus.cc:513] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.274026 1368 leader_election.cc:290] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.274555 753 tablet_service.cc:1515] Processing DeleteTablet for tablet cbaa018753d8455199f3869323fb5582 with delete_type TABLET_DATA_DELETED (replaced by tablet cd4f0857f52449e1b26fde09c6c7c9d4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.276845 1165 tablet_replica.cc:331] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.277048 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 with delete_type TABLET_DATA_DELETED (replaced by tablet d8b30c229c3445b683a5015f069eca1c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.277158 1165 raft_consensus.cc:2241] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:56.277340 1165 raft_consensus.cc:2270] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.277956 1165 ts_tablet_manager.cc:1905] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.279446 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbaa018753d8455199f3869323fb5582" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:56.288355 1170 raft_consensus.cc:383] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.288486 1170 raft_consensus.cc:738] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.288743 1170 consensus_queue.cc:260] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.289237 1170 ts_tablet_manager.cc:1428] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.017s user 0.000s sys 0.001s
W20250625 13:51:56.289172 711 leader_election.cc:343] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.194411 1167 tablet_replica.cc:331] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:56.290251 1167 raft_consensus.cc:2241] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:56.290711 1167 pending_rounds.cc:70] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:51:56.290866 1167 pending_rounds.cc:77] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520575967809536 op_type: NO_OP noop_request { }
I20250625 13:51:56.291036 1167 raft_consensus.cc:2887] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:56.291193 1167 raft_consensus.cc:2270] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.292412 1167 ts_tablet_manager.cc:1905] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.295466 1168 log.cc:1199] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/0723e91650174f5badd74f7756c5a54e
I20250625 13:51:56.296061 1168 ts_tablet_manager.cc:1939] T 0723e91650174f5badd74f7756c5a54e P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:51:56.297075 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc75a0a8421b457089cc707d9eb6b76f consensus state has no leader
I20250625 13:51:56.298772 1166 tablet_bootstrap.cc:492] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.298961 1166 ts_tablet_manager.cc:1397] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
W20250625 13:51:56.299764 581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:33015, user_credentials={real_user=slave}} blocked reactor thread for 263439us
I20250625 13:51:56.299809 1166 raft_consensus.cc:357] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:56.300393 1166 raft_consensus.cc:383] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.300490 1166 raft_consensus.cc:738] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.300772 1166 consensus_queue.cc:260] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:56.302297 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 2404e712e18146b68757bd77362c1358 (DEFAULT_TABLE table=default.Table_8f37509832f440ef94e577a34a689bfe [id=0532faedda28481f9742008aecb6bfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.302754 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2404e712e18146b68757bd77362c1358. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.303361 1166 ts_tablet_manager.cc:1428] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250625 13:51:56.303860 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 (DEFAULT_TABLE table=default.Table_020b99c8c2cb44e3982eaf49820a16d0 [id=5fcd0958c61942289b4eb28c077ceec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.305408 1013 tablet_service.cc:1468] Processing CreateTablet for tablet b14d00c93a964edd815122e3723f89e7 (DEFAULT_TABLE table=default.Table_a07c15abd0224e7d80d182ea2c891993 [id=47cd45886fc94cd7ba5cafa60838d601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.306005 1166 tablet_bootstrap.cc:492] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:51:56.306384 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46316eb725bc47e695af476c0b185504 consensus state has no leader
I20250625 13:51:56.307071 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b14d00c93a964edd815122e3723f89e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.307574 1028 tablet_service.cc:1468] Processing CreateTablet for tablet ab7202a8761444a0a731993d890fcdeb (DEFAULT_TABLE table=default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.308034 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (DEFAULT_TABLE table=default.Table_40ca12b3cc9a4632912f3508933960e2 [id=25a7472e96b4448683b5fcf35e540048]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.307415 1011 tablet_service.cc:1468] Processing CreateTablet for tablet ac4d368347444846880954ced9651404 (DEFAULT_TABLE table=default.Table_581fcddbe9bb428d84556e6156637b4a [id=3a7ba6e0a96c4f3fb656f9ac4f6ee82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.308959 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 7108932fa8a241819ca90674a64b285e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.309208 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5786bb7716ef44689ef9862fc3ce17e9 with delete_type TABLET_DATA_DELETED (replaced by tablet d8b30c229c3445b683a5015f069eca1c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.309314 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a21873ae3bbd4a1aaf287e80fe14c12e" } modify_external_catalogs: true
I20250625 13:51:56.309414 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 13620f6ca65342b4ad7f1ba4b437bde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.309516 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a21873ae3bbd4a1aaf287e80fe14c12e" } modify_external_catalogs: true
I20250625 13:51:56.309595 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a23b08a03b48eaa051f670eb7dac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.310269 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79beb23eebf54fd9951ea5a33fcfb0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.310628 1166 tablet_bootstrap.cc:654] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.311151 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d368347444846880954ced9651404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.312053 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7202a8761444a0a731993d890fcdeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.313045 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b549bc8cead4af4a72f41f51655602c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.289549 1170 tablet_bootstrap.cc:492] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:56.318009 1170 tablet_bootstrap.cc:654] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.319921 1165 ts_tablet_manager.cc:1918] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.319967 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_56e4bf4e04984c3db6effdb227c7a75d" } new_table_name: "default.Table_c7891a214410484c907ee024a3f40331" modify_external_catalogs: true
I20250625 13:51:56.320880 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.321363 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 430c3bed360b4f32855a8adb99076608 (DEFAULT_TABLE table=default.Table_8319fc497f694699988a5dfd8de276e0 [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.321557 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3df36037c9584e4c94d313cdf1402fdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.321825 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 430c3bed360b4f32855a8adb99076608. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.322116 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3df36037c9584e4c94d313cdf1402fdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.323403 1165 log.cc:1199] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ce84f74470ad428abb2113efa170ca24
I20250625 13:51:56.324018 1165 ts_tablet_manager.cc:1939] T ce84f74470ad428abb2113efa170ca24 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.324265 1046 raft_consensus.cc:3058] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.324975 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 206) took 1410 ms. Trace:
I20250625 13:51:56.325153 1165 rpcz_store.cc:276] 0625 13:51:54.914265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.914325 (+ 60us) service_pool.cc:225] Handling call
0625 13:51:56.272139 (+1357814us) ts_tablet_manager.cc:1145] Deleting tablet ce84f74470ad428abb2113efa170ca24
0625 13:51:56.276733 (+ 4594us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.281894 (+ 5161us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.319755 (+ 37861us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.324937 (+ 5182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357615}
I20250625 13:51:56.325814 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 207) took 1357 ms. Trace:
I20250625 13:51:56.325825 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ce84f74470ad428abb2113efa170ca24 (table default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]) successfully deleted
I20250625 13:51:56.325999 1165 rpcz_store.cc:276] 0625 13:51:54.967975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.970444 (+ 2469us) service_pool.cc:225] Handling call
0625 13:51:56.325658 (+1355214us) ts_tablet_manager.cc:1145] Deleting tablet 14a12a5453e647b9946ca19e0ffb14ee
0625 13:51:56.325670 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.325778 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355002}
I20250625 13:51:56.326553 1165 tablet_replica.cc:331] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:51:56.326519 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 14a12a5453e647b9946ca19e0ffb14ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14a12a5453e647b9946ca19e0ffb14ee already in progress: creating tablet
I20250625 13:51:56.327003 1165 raft_consensus.cc:2241] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:56.327580 1165 pending_rounds.cc:70] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:56.328119 1165 pending_rounds.cc:77] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520575047610368 op_type: NO_OP noop_request { }
I20250625 13:51:56.330289 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.330894 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a7b72c2c464421eacfcf64008c3aea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.331601 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 36b151e200e1430c99ac194bc55881c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.333742 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 (DEFAULT_TABLE table=default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.334271 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f5b36ddb474ed6903c14876b236ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.279095 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbaa018753d8455199f3869323fb5582" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:56.335268 1165 raft_consensus.cc:2887] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:56.335423 1165 raft_consensus.cc:2270] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.335523 914 raft_consensus.cc:2466] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:56.261024 915 raft_consensus.cc:2466] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:56.336179 1166 tablet_bootstrap.cc:492] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.336370 711 leader_election.cc:304] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:56.336545 1166 ts_tablet_manager.cc:1397] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250625 13:51:56.336777 1368 raft_consensus.cc:2802] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:56.336894 1368 raft_consensus.cc:491] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:56.336987 1368 raft_consensus.cc:3058] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.337469 1166 raft_consensus.cc:357] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.337670 1166 raft_consensus.cc:383] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.337781 1166 raft_consensus.cc:738] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.338034 1166 consensus_queue.cc:260] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:56.299103 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 362128us
I20250625 13:51:56.340454 1368 raft_consensus.cc:513] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.341004 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbaa018753d8455199f3869323fb5582" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:56.341128 1368 leader_election.cc:290] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.341154 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:56.341267 776 raft_consensus.cc:2391] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate affa1c3035b940e88ca143be7d35840d in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:51:56.341522 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5786bb7716ef44689ef9862fc3ce17e9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:56.341687 777 raft_consensus.cc:2466] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:56.341779 711 leader_election.cc:304] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:56.307667 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 7108932fa8a241819ca90674a64b285e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.342182 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf5a6951aba84f5fa8f6d76a2b7fcf2d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:56.308473 878 tablet_service.cc:1468] Processing CreateTablet for tablet ac4d368347444846880954ced9651404 (DEFAULT_TABLE table=default.Table_581fcddbe9bb428d84556e6156637b4a [id=3a7ba6e0a96c4f3fb656f9ac4f6ee82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.309033 895 tablet_service.cc:1468] Processing CreateTablet for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (DEFAULT_TABLE table=default.Table_40ca12b3cc9a4632912f3508933960e2 [id=25a7472e96b4448683b5fcf35e540048]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.343152 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d368347444846880954ced9651404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.344185 851 leader_election.cc:304] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:51:56.344874 851 leader_election.cc:343] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.310721 886 tablet_service.cc:1468] Processing CreateTablet for tablet 2404e712e18146b68757bd77362c1358 (DEFAULT_TABLE table=default.Table_8f37509832f440ef94e577a34a689bfe [id=0532faedda28481f9742008aecb6bfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.310904 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 13620f6ca65342b4ad7f1ba4b437bde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.311396 879 tablet_service.cc:1468] Processing CreateTablet for tablet b14d00c93a964edd815122e3723f89e7 (DEFAULT_TABLE table=default.Table_a07c15abd0224e7d80d182ea2c891993 [id=47cd45886fc94cd7ba5cafa60838d601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.312017 891 tablet_service.cc:1468] Processing CreateTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 (DEFAULT_TABLE table=default.Table_020b99c8c2cb44e3982eaf49820a16d0 [id=5fcd0958c61942289b4eb28c077ceec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.312775 880 tablet_service.cc:1468] Processing CreateTablet for tablet 430c3bed360b4f32855a8adb99076608 (DEFAULT_TABLE table=default.Table_8319fc497f694699988a5dfd8de276e0 [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.320648 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.327881 1307 consensus_queue.cc:1035] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:56.342624 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b549bc8cead4af4a72f41f51655602c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:56.346851 777 raft_consensus.cc:2466] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:56.343655 1166 ts_tablet_manager.cc:1428] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 13:51:56.347084 1368 raft_consensus.cc:2802] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:56.347393 1166 tablet_bootstrap.cc:492] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.347391 1368 raft_consensus.cc:695] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:56.347798 1368 consensus_queue.cc:237] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.349742 1166 tablet_bootstrap.cc:654] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.350236 1307 raft_consensus.cc:3058] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.351339 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c1a98dc40684954a602567cf3476879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.344211 1165 ts_tablet_manager.cc:1905] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.353438 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79beb23eebf54fd9951ea5a33fcfb0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.353981 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c3f75eb1454e43e2a012dad91ae07d37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.328598 881 tablet_service.cc:1468] Processing CreateTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 (DEFAULT_TABLE table=default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.329813 888 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:51:56.354547 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3f75eb1454e43e2a012dad91ae07d37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.331424 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 36b151e200e1430c99ac194bc55881c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.345865 753 tablet_service.cc:1468] Processing CreateTablet for tablet 024f6cb363594420b0d5480fa79d156b (DEFAULT_TABLE table=default.Table_0892d2b459334a8bac680a69f2be033e [id=3eae0d10bcb247e0a318d6ac81e8c5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.346556 745 tablet_service.cc:1468] Processing CreateTablet for tablet 042359eacaa2451abc0fa6ed447a2905 (DEFAULT_TABLE table=default.Table_f0caf6cb5cc84df9b75703d4a12eb505 [id=426662edb72f4d1aa3e0aa1ee934e62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.355291 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 024f6cb363594420b0d5480fa79d156b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.355827 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 042359eacaa2451abc0fa6ed447a2905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.356374 1307 raft_consensus.cc:2747] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:56.356617 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbaa018753d8455199f3869323fb5582" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:56.363375 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2404e712e18146b68757bd77362c1358. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.366111 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2404e712e18146b68757bd77362c1358 consensus state has no leader
I20250625 13:51:56.367444 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 430c3bed360b4f32855a8adb99076608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.368791 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b14d00c93a964edd815122e3723f89e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.369411 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.369943 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.370990 1165 ts_tablet_manager.cc:1918] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:56.371119 1165 log.cc:1199] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d0cf3d979369419882a2ba2083933d48
I20250625 13:51:56.332978 877 tablet_service.cc:1468] Processing CreateTablet for tablet ab7202a8761444a0a731993d890fcdeb (DEFAULT_TABLE table=default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.373623 1165 ts_tablet_manager.cc:1939] T d0cf3d979369419882a2ba2083933d48 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.345173 851 leader_election.cc:304] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:51:56.377009 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abd6f9121a9d4b39adde1b880a875e23 consensus state has no leader
I20250625 13:51:56.374423 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 208) took 1401 ms. Trace:
I20250625 13:51:56.379478 1165 rpcz_store.cc:276] 0625 13:51:54.973030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.973539 (+ 509us) service_pool.cc:225] Handling call
0625 13:51:56.326482 (+1352943us) ts_tablet_manager.cc:1145] Deleting tablet d0cf3d979369419882a2ba2083933d48
0625 13:51:56.326490 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.354398 (+ 27908us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.370864 (+ 16466us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.374389 (+ 3525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1352736}
I20250625 13:51:56.379920 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 921163dc5f454c9a97e72b005f9dba19 (table default.Table_b13c5d5237f044c9bad038533aecc142 [id=96df25b43f664e9ebc958ff801d229b5]) successfully deleted
I20250625 13:51:56.381243 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d0cf3d979369419882a2ba2083933d48 (table default.Table_ac5c66505c43484ca1990febfc2a5bf9 [id=eb254e00e07047a09c1924a6cab20e4d]) successfully deleted
W20250625 13:51:56.384737 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ba1338d0562418eadabdb8dbcaedee2 has no consensus state
I20250625 13:51:56.384757 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbaa018753d8455199f3869323fb5582" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:56.385087 1165 tablet_replica.cc:331] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.386309 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_068bedd508da462681c0138ddd8964e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.386807 1165 raft_consensus.cc:2241] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:56.386993 1165 raft_consensus.cc:2270] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:56.386988 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_068bedd508da462681c0138ddd8964e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.387580 1165 ts_tablet_manager.cc:1905] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.386273 983 leader_election.cc:304] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:56.389411 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 024f6cb363594420b0d5480fa79d156b (DEFAULT_TABLE table=default.Table_0892d2b459334a8bac680a69f2be033e [id=3eae0d10bcb247e0a318d6ac81e8c5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.393549 1302 raft_consensus.cc:491] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.393847 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 7108932fa8a241819ca90674a64b285e from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:56.393796 1302 raft_consensus.cc:513] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.394565 1302 leader_election.cc:290] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.395444 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 024f6cb363594420b0d5480fa79d156b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.398357 1166 tablet_bootstrap.cc:492] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.398543 1166 ts_tablet_manager.cc:1397] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.007s
W20250625 13:51:56.389235 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 47721.5us
I20250625 13:51:56.399572 1166 raft_consensus.cc:357] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.399802 1166 raft_consensus.cc:383] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.399884 1166 raft_consensus.cc:738] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.400151 1166 consensus_queue.cc:260] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.400674 1166 ts_tablet_manager.cc:1428] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:56.401019 1166 tablet_bootstrap.cc:492] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.403194 1166 tablet_bootstrap.cc:654] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:56.404350 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 47021.4us
I20250625 13:51:56.406790 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 01ba67b500b646d7aa55315f7ee23ed6
I20250625 13:51:56.407218 591 catalog_manager.cc:6652] ReplaceTablet: tablet 01ba67b500b646d7aa55315f7ee23ed6 deleted and replaced by tablet 32143e2497ee4d1b8c2e7cbfc6160ef4
I20250625 13:51:56.410310 983 leader_election.cc:304] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:56.410611 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 with delete_type TABLET_DATA_DELETED (replaced by tablet 32143e2497ee4d1b8c2e7cbfc6160ef4) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:56.411465 1015 consensus_meta.cc:287] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent flushing consensus metadata: real 0.543s user 0.007s sys 0.000s
I20250625 13:51:56.411687 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_221b43b171de43b19b753208d05a4e40" } modify_external_catalogs: true
I20250625 13:51:56.411923 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_221b43b171de43b19b753208d05a4e40" } modify_external_catalogs: true
I20250625 13:51:56.413343 1307 raft_consensus.cc:2747] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:56.393540 711 leader_election.cc:343] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.416682 1165 ts_tablet_manager.cc:1918] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.416838 1165 log.cc:1199] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/99090fee41d44946b6243b1b4bff3635
I20250625 13:51:56.417227 1165 ts_tablet_manager.cc:1939] T 99090fee41d44946b6243b1b4bff3635 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:51:56.417861 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 2404e712e18146b68757bd77362c1358 (table default.Table_9b1b976f4c234b97923637678b6508f9 [id=0532faedda28481f9742008aecb6bfd3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2404e712e18146b68757bd77362c1358 consensus state has no leader
I20250625 13:51:56.421170 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 209) took 1447 ms. Trace:
I20250625 13:51:56.421209 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 991bb428def545fd979e9f1a80ce4d18
I20250625 13:51:56.421284 1165 rpcz_store.cc:276] 0625 13:51:54.973160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.976160 (+ 3000us) service_pool.cc:225] Handling call
0625 13:51:56.385021 (+1408861us) ts_tablet_manager.cc:1145] Deleting tablet 99090fee41d44946b6243b1b4bff3635
0625 13:51:56.385032 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.407951 (+ 22919us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.416560 (+ 8609us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.421113 (+ 4553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408070}
I20250625 13:51:56.421906 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a23b08a03b48eaa051f670eb7dac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.422616 1166 tablet_bootstrap.cc:492] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.422788 1166 ts_tablet_manager.cc:1397] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250625 13:51:56.423285 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 991bb428def545fd979e9f1a80ce4d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.423558 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 99090fee41d44946b6243b1b4bff3635 (table default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]) successfully deleted
I20250625 13:51:56.423724 1166 raft_consensus.cc:357] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.423913 1166 raft_consensus.cc:383] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.424022 1166 raft_consensus.cc:738] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.424288 1166 consensus_queue.cc:260] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.424401 915 raft_consensus.cc:2466] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:56.424796 1166 ts_tablet_manager.cc:1428] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:51:56.425160 1166 tablet_bootstrap.cc:492] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.425421 851 leader_election.cc:304] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:56.425817 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13620f6ca65342b4ad7f1ba4b437bde3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:56.426019 914 raft_consensus.cc:2466] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:56.426350 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f5b36ddb474ed6903c14876b236ba8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.427738 851 leader_election.cc:343] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 67441795e22b445a90c88d76aa519ea5
I20250625 13:51:56.428469 1307 raft_consensus.cc:2802] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:56.428050 851 leader_election.cc:304] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:56.429226 1424 raft_consensus.cc:2747] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:56.429539 851 leader_election.cc:343] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a21335a06ec04da29b3351554c75f03c
I20250625 13:51:56.429795 851 leader_election.cc:304] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:56.430102 1424 raft_consensus.cc:2747] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:56.430552 1165 tablet_replica.cc:331] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.430579 851 leader_election.cc:304] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:56.430768 1165 raft_consensus.cc:2241] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:56.430968 1165 raft_consensus.cc:2270] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.431386 1424 raft_consensus.cc:2802] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:56.431499 1424 raft_consensus.cc:491] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:56.431509 1165 ts_tablet_manager.cc:1905] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.431821 1424 raft_consensus.cc:3058] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.435101 1424 raft_consensus.cc:513] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:56.435915 1424 leader_election.cc:290] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:56.436015 1166 tablet_bootstrap.cc:654] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.436379 1307 raft_consensus.cc:491] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:56.436535 1307 raft_consensus.cc:3058] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.440668 1307 raft_consensus.cc:513] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.441751 1307 leader_election.cc:290] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:56.443418 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5786bb7716ef44689ef9862fc3ce17e9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:56.443619 776 raft_consensus.cc:3058] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.444180 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ad4e07371e3b42c5b05954f9537fd6b7" } modify_external_catalogs: true
I20250625 13:51:56.444202 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_42f697776fd04dc3ab9be41c229289fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.444416 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ad4e07371e3b42c5b05954f9537fd6b7" } modify_external_catalogs: true
W20250625 13:51:56.444824 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42f697776fd04dc3ab9be41c229289fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.444942 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae" } modify_external_catalogs: true
I20250625 13:51:56.445123 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae" } modify_external_catalogs: true
I20250625 13:51:56.445698 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c6d92fec1b3f4f368c408fc302731df0" } modify_external_catalogs: true
I20250625 13:51:56.445834 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c6d92fec1b3f4f368c408fc302731df0" } modify_external_catalogs: true
I20250625 13:51:56.446377 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b549bc8cead4af4a72f41f51655602c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:56.446588 777 raft_consensus.cc:3058] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.446740 776 raft_consensus.cc:2466] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:56.447782 851 leader_election.cc:304] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:56.448115 1424 consensus_queue.cc:1035] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:56.448633 1424 raft_consensus.cc:2802] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:56.448776 1424 raft_consensus.cc:695] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:56.449070 1424 consensus_queue.cc:237] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.449854 777 raft_consensus.cc:2466] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:56.455161 1165 ts_tablet_manager.cc:1918] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.455413 851 leader_election.cc:304] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:56.455724 1424 raft_consensus.cc:2802] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:56.456048 1424 raft_consensus.cc:695] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:56.456355 1424 consensus_queue.cc:237] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:56.459023 1166 tablet_bootstrap.cc:492] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.459297 1166 ts_tablet_manager.cc:1397] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.005s
I20250625 13:51:56.460161 1166 raft_consensus.cc:357] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.460606 1166 raft_consensus.cc:383] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.460731 1166 raft_consensus.cc:738] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.461045 1166 consensus_queue.cc:260] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.461633 1166 ts_tablet_manager.cc:1428] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:51:56.461745 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c2fa8e9e8b04497eb6ba7bec9a7379d9
I20250625 13:51:56.462129 1166 tablet_bootstrap.cc:492] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.463932 1166 tablet_bootstrap.cc:654] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.464097 1165 log.cc:1199] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/b11b3157dd764f229831c3b8417460c6
I20250625 13:51:56.465914 1165 ts_tablet_manager.cc:1939] T b11b3157dd764f229831c3b8417460c6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.466709 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 210) took 1479 ms. Trace:
I20250625 13:51:56.468587 1165 rpcz_store.cc:276] 0625 13:51:54.987701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.989857 (+ 2156us) service_pool.cc:225] Handling call
0625 13:51:56.430468 (+1440611us) ts_tablet_manager.cc:1145] Deleting tablet b11b3157dd764f229831c3b8417460c6
0625 13:51:56.430476 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.439728 (+ 9252us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.455005 (+ 15277us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.466674 (+ 11669us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1435466}
I20250625 13:51:56.469213 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 211) took 1451 ms. Trace:
I20250625 13:51:56.469734 1165 rpcz_store.cc:276] 0625 13:51:55.018053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.018119 (+ 66us) service_pool.cc:225] Handling call
0625 13:51:56.469077 (+1450958us) ts_tablet_manager.cc:1145] Deleting tablet 389d81329b8d430a9098b0630612fdcb
0625 13:51:56.469086 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.469182 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450706}
W20250625 13:51:56.470739 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 389d81329b8d430a9098b0630612fdcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 389d81329b8d430a9098b0630612fdcb already in progress: creating tablet
I20250625 13:51:56.471135 1165 tablet_replica.cc:331] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.472280 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3be2a3bc86fd4fd993dbfa89af6b7b97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.472548 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 134) took 2278 ms. Trace:
I20250625 13:51:56.472766 1165 raft_consensus.cc:2241] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:56.472802 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3be2a3bc86fd4fd993dbfa89af6b7b97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.472684 1168 rpcz_store.cc:276] 0625 13:51:54.115129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.118526 (+ 3397us) service_pool.cc:225] Handling call
0625 13:51:55.972881 (+1854355us) ts_tablet_manager.cc:1145] Deleting tablet 0723e91650174f5badd74f7756c5a54e
0625 13:51:55.972891 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.218859 (+245968us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.249533 (+ 30674us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.393238 (+143705us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1854132}
I20250625 13:51:56.472922 1165 raft_consensus.cc:2270] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.473543 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7202a8761444a0a731993d890fcdeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.473559 1165 ts_tablet_manager.cc:1905] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.483028 1166 tablet_bootstrap.cc:492] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.483208 1166 ts_tablet_manager.cc:1397] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250625 13:51:56.483901 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8d37b99f76eb49c8af370ba05e4015aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.484275 1166 raft_consensus.cc:357] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.484441 1166 raft_consensus.cc:383] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:51:56.484455 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d37b99f76eb49c8af370ba05e4015aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.484534 1166 raft_consensus.cc:738] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.484817 1166 consensus_queue.cc:260] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.485317 1166 ts_tablet_manager.cc:1428] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:56.485623 1166 tablet_bootstrap.cc:492] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.487819 1166 tablet_bootstrap.cc:654] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.488813 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b549bc8cead4af4a72f41f51655602c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:56.490155 1168 tablet_replica.cc:331] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:56.490478 1168 raft_consensus.cc:2241] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:56.490782 1168 raft_consensus.cc:2270] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.491369 1168 ts_tablet_manager.cc:1905] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.492213 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5513b6e4edc14bdea44f9e8562b4d70e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.492720 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5513b6e4edc14bdea44f9e8562b4d70e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.493532 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13620f6ca65342b4ad7f1ba4b437bde3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:56.498955 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_776dedc025c447b3a2709df16fa5d505" } new_table_name: "default.Table_9e22da664adb404cad353db7745d0110" modify_external_catalogs: true
I20250625 13:51:56.501287 1165 ts_tablet_manager.cc:1918] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.501466 1165 log.cc:1199] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/921163dc5f454c9a97e72b005f9dba19
I20250625 13:51:56.501878 1165 ts_tablet_manager.cc:1939] T 921163dc5f454c9a97e72b005f9dba19 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.503031 751 tablet_service.cc:1468] Processing CreateTablet for tablet 20c12f55ed5b4425aea1bf45cdc0658e (DEFAULT_TABLE table=default.Table_6b0fc9a5299d475aa3ac845d06953e17 [id=06980d9c71814771ae6ccb685b4e54c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.503057 753 tablet_service.cc:1468] Processing CreateTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 (DEFAULT_TABLE table=default.Table_de6ae2243e604e0a8094e43c5b76ca27 [id=6de15b35e99646eb8eba9c11eea579c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.503504 745 tablet_service.cc:1468] Processing CreateTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 (DEFAULT_TABLE table=default.Table_c7891a214410484c907ee024a3f40331 [id=21a2a642fff348518e79d4e4db1f32ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.503779 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c12f55ed5b4425aea1bf45cdc0658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.504112 743 tablet_service.cc:1468] Processing CreateTablet for tablet 74884bf313724d678d3b5f5869f09f97 (DEFAULT_TABLE table=default.Table_25d9c73fdf1b43b787345769f54915d6 [id=79ace8a081c447a5b4c44b5911a906c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.504503 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74884bf313724d678d3b5f5869f09f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.507620 755 tablet_service.cc:1468] Processing CreateTablet for tablet 5e280bd329764924a6fd42c591888ed0 (DEFAULT_TABLE table=default.Table_2211bebab65640d09cfecd34b8eaaa40 [id=3d206a6b57d34bf599390f47c855c11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.507735 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c5fc63828d9408b9cf774a966f43cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.507997 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e280bd329764924a6fd42c591888ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.507958 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d2296c79c2f949fca8a57149e361225e" } new_table_name: "default.Table_f51c5c5fe5394b598404b1944ee7d0b5" modify_external_catalogs: true
I20250625 13:51:56.511663 750 tablet_service.cc:1468] Processing CreateTablet for tablet 490c8902efeb46c99543747a8787c1c7 (DEFAULT_TABLE table=default.Table_dacf434b035e41c293c9fb0057989436 [id=a0e4a71b13d94a69affc3cc9e0cb5066]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.512044 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490c8902efeb46c99543747a8787c1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.515770 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba1338d0562418eadabdb8dbcaedee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.520541 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 (DEFAULT_TABLE table=default.Table_3462a32d0970423c8538a79dae984471 [id=60ccb9d481c14fd68670e5701ce76b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.521683 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9514bbbda0ba4329a18b0a42e635c6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.521160 753 tablet_service.cc:1468] Processing CreateTablet for tablet a29708fb688a4a65b84ae2564a040b4c (DEFAULT_TABLE table=default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.523599 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29708fb688a4a65b84ae2564a040b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.528250 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 212) took 1477 ms. Trace:
I20250625 13:51:56.531760 1165 rpcz_store.cc:276] 0625 13:51:55.051166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.051217 (+ 51us) service_pool.cc:225] Handling call
0625 13:51:56.471071 (+1419854us) ts_tablet_manager.cc:1145] Deleting tablet 921163dc5f454c9a97e72b005f9dba19
0625 13:51:56.471081 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.489608 (+ 18527us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.501159 (+ 11551us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.528181 (+ 27022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419600}
I20250625 13:51:56.535429 1165 tablet_replica.cc:331] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.536379 1165 raft_consensus.cc:2241] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:56.536967 1165 raft_consensus.cc:2270] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.537359 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 921163dc5f454c9a97e72b005f9dba19 (table default.Table_b13c5d5237f044c9bad038533aecc142 [id=96df25b43f664e9ebc958ff801d229b5]) successfully deleted
I20250625 13:51:56.538290 1165 ts_tablet_manager.cc:1905] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.537703 1166 tablet_bootstrap.cc:492] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.538723 1166 ts_tablet_manager.cc:1397] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.000s
I20250625 13:51:56.539518 1166 raft_consensus.cc:357] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.539789 1166 raft_consensus.cc:383] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.539888 1166 raft_consensus.cc:738] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.540136 1166 consensus_queue.cc:260] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.540865 1166 ts_tablet_manager.cc:1428] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:56.541181 1166 tablet_bootstrap.cc:492] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.547000 1166 tablet_bootstrap.cc:654] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.555732 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3ac6f01366104d75a5bf908bc40c3f4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.556351 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ac6f01366104d75a5bf908bc40c3f4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.558399 711 leader_election.cc:304] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:56.558858 1302 raft_consensus.cc:2802] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:56.559060 1302 raft_consensus.cc:695] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:56.559461 1302 consensus_queue.cc:237] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.559686 1165 ts_tablet_manager.cc:1918] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.559818 1165 log.cc:1199] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/cf5a6951aba84f5fa8f6d76a2b7fcf2d
I20250625 13:51:56.560228 1165 ts_tablet_manager.cc:1939] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.560652 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3b0037d09796455cbaaadfaa9c010673
I20250625 13:51:56.562955 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b0037d09796455cbaaadfaa9c010673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.565114 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bb1c7a85da844d34be805843bc60d350
I20250625 13:51:56.567198 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 643928ca4fc94ce58431a0525f2abb99
I20250625 13:51:56.568131 753 tablet_service.cc:1515] Processing DeleteTablet for tablet bb1c7a85da844d34be805843bc60d350 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.568666 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fdfdf918b84c4bc7874ae0a62279c7c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.569531 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_020b99c8c2cb44e3982eaf49820a16d0" } new_table_name: "default.Table_1dd56390e1b04565a7850f5e960cddad" modify_external_catalogs: true
W20250625 13:51:56.570796 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdfdf918b84c4bc7874ae0a62279c7c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.571054 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 643928ca4fc94ce58431a0525f2abb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.574362 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 213) took 1346 ms. Trace:
I20250625 13:51:56.575294 1165 rpcz_store.cc:276] 0625 13:51:55.227697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.241781 (+ 14084us) service_pool.cc:225] Handling call
0625 13:51:56.535331 (+1293550us) ts_tablet_manager.cc:1145] Deleting tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d
0625 13:51:56.535340 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.552662 (+ 17322us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.559514 (+ 6852us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.574319 (+ 14805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233949}
I20250625 13:51:56.576098 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d (table default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e [id=2a70f0ade87f4f34bcbe23295f23234b]) successfully deleted
I20250625 13:51:56.576279 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 216) took 1237 ms. Trace:
I20250625 13:51:56.576732 1165 rpcz_store.cc:276] 0625 13:51:55.338951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.339007 (+ 56us) service_pool.cc:225] Handling call
0625 13:51:56.576104 (+1237097us) ts_tablet_manager.cc:1145] Deleting tablet 04479dfb34584f0891c604ef72bf93b5
0625 13:51:56.576116 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.576242 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236844}
I20250625 13:51:56.577491 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 217) took 1206 ms. Trace:
W20250625 13:51:56.577531 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 04479dfb34584f0891c604ef72bf93b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04479dfb34584f0891c604ef72bf93b5 already in progress: creating tablet
I20250625 13:51:56.578131 1165 rpcz_store.cc:276] 0625 13:51:55.371402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.372887 (+ 1485us) service_pool.cc:225] Handling call
0625 13:51:56.577360 (+1204473us) ts_tablet_manager.cc:1145] Deleting tablet 07e281786e6e4a5d93b94b85b486655d
0625 13:51:56.577368 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.577459 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1204263}
I20250625 13:51:56.578946 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 218) took 1207 ms. Trace:
W20250625 13:51:56.579437 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 07e281786e6e4a5d93b94b85b486655d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07e281786e6e4a5d93b94b85b486655d already in progress: creating tablet
I20250625 13:51:56.579381 1165 rpcz_store.cc:276] 0625 13:51:55.371533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.372828 (+ 1295us) service_pool.cc:225] Handling call
0625 13:51:56.578796 (+1205968us) ts_tablet_manager.cc:1145] Deleting tablet 6771d8e180fe48569a0e016db0cc9b51
0625 13:51:56.578805 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.578913 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1204427}
W20250625 13:51:56.579821 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6771d8e180fe48569a0e016db0cc9b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6771d8e180fe48569a0e016db0cc9b51 already in progress: creating tablet
I20250625 13:51:56.580092 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 219) took 1190 ms. Trace:
I20250625 13:51:56.580502 1165 rpcz_store.cc:276] 0625 13:51:55.389486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.389549 (+ 63us) service_pool.cc:225] Handling call
0625 13:51:56.579969 (+1190420us) ts_tablet_manager.cc:1145] Deleting tablet 85c72f9ae92a4537979fb479eb4467c4
0625 13:51:56.579978 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.580062 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190167}
W20250625 13:51:56.581162 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 85c72f9ae92a4537979fb479eb4467c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85c72f9ae92a4537979fb479eb4467c4 already in progress: creating tablet
I20250625 13:51:56.581303 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 220) took 1165 ms. Trace:
I20250625 13:51:56.581699 1165 rpcz_store.cc:276] 0625 13:51:55.416094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.416146 (+ 52us) service_pool.cc:225] Handling call
0625 13:51:56.581188 (+1165042us) ts_tablet_manager.cc:1145] Deleting tablet f00e72e103f443e886f0df690cff4b6f
0625 13:51:56.581197 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.581275 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164811}
I20250625 13:51:56.581806 1171 tablet_bootstrap.cc:492] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:56.582012 1171 ts_tablet_manager.cc:1397] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.622s user 0.001s sys 0.007s
I20250625 13:51:56.582733 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_10094a18787042588bd628e824da998f" } new_table_name: "default.Table_baa360c7bf404bb79b9e32c5d95dc8b5" modify_external_catalogs: true
I20250625 13:51:56.582717 1166 tablet_bootstrap.cc:492] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.582998 1166 ts_tablet_manager.cc:1397] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.000s
I20250625 13:51:56.583802 1166 raft_consensus.cc:357] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.583974 1166 raft_consensus.cc:383] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.584065 1166 raft_consensus.cc:738] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.584378 1166 consensus_queue.cc:260] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.585062 1166 ts_tablet_manager.cc:1428] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:56.585323 1166 tablet_bootstrap.cc:492] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:51:56.585978 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8063122ea7b24b6790f5c3cc884e09be has no consensus state
I20250625 13:51:56.586539 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8172b8ab4c3d4d8ca80409212c6866ff" } new_table_name: "default.Table_47598e6b1611413784a8c9ab9d1c0a67" modify_external_catalogs: true
W20250625 13:51:56.586865 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f00e72e103f443e886f0df690cff4b6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f00e72e103f443e886f0df690cff4b6f already in progress: creating tablet
I20250625 13:51:56.587512 1166 tablet_bootstrap.cc:654] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.588532 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 221) took 1110 ms. Trace:
I20250625 13:51:56.588697 1165 rpcz_store.cc:276] 0625 13:51:55.478436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.478618 (+ 182us) service_pool.cc:225] Handling call
0625 13:51:56.588416 (+1109798us) ts_tablet_manager.cc:1145] Deleting tablet c666ff2d6b9b4ed4a18e370826c143fc
0625 13:51:56.588424 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.588502 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109573}
I20250625 13:51:56.589157 1165 tablet_replica.cc:331] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.589133 1424 consensus_queue.cc:1035] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:51:56.589345 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c666ff2d6b9b4ed4a18e370826c143fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c666ff2d6b9b4ed4a18e370826c143fc already in progress: creating tablet
I20250625 13:51:56.589496 1165 raft_consensus.cc:2241] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:56.589762 1165 pending_rounds.cc:70] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:56.589890 1165 pending_rounds.cc:77] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520577787473920 op_type: NO_OP noop_request { }
I20250625 13:51:56.590024 1165 raft_consensus.cc:2887] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:56.590193 1165 raft_consensus.cc:2270] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.590873 1165 ts_tablet_manager.cc:1905] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:56.597126 849 leader_election.cc:343] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.601179 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5786bb7716ef44689ef9862fc3ce17e9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:56.601449 1047 raft_consensus.cc:3058] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:56.603935 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 455884b62ee44b6da53543093f721027 consensus state has no leader
I20250625 13:51:56.604657 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_25705d30e7034c57be006958705ab688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.605118 1047 raft_consensus.cc:2466] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
W20250625 13:51:56.605297 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25705d30e7034c57be006958705ab688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.605456 1165 ts_tablet_manager.cc:1918] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:56.605602 1165 log.cc:1199] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:56.606020 1165 ts_tablet_manager.cc:1939] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.610165 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 222) took 1131 ms. Trace:
I20250625 13:51:56.610747 1165 rpcz_store.cc:276] 0625 13:51:55.478560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.478887 (+ 327us) service_pool.cc:225] Handling call
0625 13:51:56.589085 (+1110198us) ts_tablet_manager.cc:1145] Deleting tablet 09a8e15d57f14702ba0b5b52a94fc7c4
0625 13:51:56.589094 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.594266 (+ 5172us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.605325 (+ 11059us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.610126 (+ 4801us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110051}
I20250625 13:51:56.611661 1165 tablet_replica.cc:331] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.611629 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 09a8e15d57f14702ba0b5b52a94fc7c4 (table default.Table_abfcbd249e4a4189aba6f16b5a66a0e7 [id=7d3d8e359ea945cc9495eb2454e1d85b]) successfully deleted
I20250625 13:51:56.612166 1165 raft_consensus.cc:2241] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:56.612712 1165 pending_rounds.cc:70] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:56.613230 1165 pending_rounds.cc:77] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520577554845696 op_type: NO_OP noop_request { }
I20250625 13:51:56.613576 1165 raft_consensus.cc:2887] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:56.613902 1165 raft_consensus.cc:2270] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:56.612656 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 54266.9us
I20250625 13:51:56.614425 711 leader_election.cc:304] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:56.615139 1368 raft_consensus.cc:2802] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:56.615314 1368 raft_consensus.cc:491] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:56.616636 1368 raft_consensus.cc:3058] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:56.620321 711 leader_election.cc:343] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.621904 1165 ts_tablet_manager.cc:1905] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.623111 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_de6ae2243e604e0a8094e43c5b76ca27" } modify_external_catalogs: true
I20250625 13:51:56.623342 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_de6ae2243e604e0a8094e43c5b76ca27" } modify_external_catalogs: true
W20250625 13:51:56.624022 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38a23b08a03b48eaa051f670eb7dac65 consensus state has no leader
I20250625 13:51:56.624742 1368 raft_consensus.cc:513] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.625924 1368 leader_election.cc:290] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.626641 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13620f6ca65342b4ad7f1ba4b437bde3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:56.626905 914 raft_consensus.cc:3058] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.628801 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 3 got 0
I20250625 13:51:56.629266 1171 raft_consensus.cc:357] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.629444 1171 raft_consensus.cc:383] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.629537 1171 raft_consensus.cc:738] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.629932 914 raft_consensus.cc:2466] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:56.630214 1171 consensus_queue.cc:260] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.630435 711 leader_election.cc:304] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:56.631465 1368 raft_consensus.cc:2802] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:56.631922 1368 raft_consensus.cc:695] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:56.632373 1368 consensus_queue.cc:237] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.637226 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_845a00e4f2b34e549af0d49c36c7899d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.638162 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_845a00e4f2b34e549af0d49c36c7899d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.641677 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_531f237d57114b2483ae3cf8d4b94659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.642968 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_531f237d57114b2483ae3cf8d4b94659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.645002 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 042359eacaa2451abc0fa6ed447a2905 (DEFAULT_TABLE table=default.Table_f0caf6cb5cc84df9b75703d4a12eb505 [id=426662edb72f4d1aa3e0aa1ee934e62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.646229 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 042359eacaa2451abc0fa6ed447a2905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.647622 1165 ts_tablet_manager.cc:1918] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:56.647881 1165 log.cc:1199] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/7fe053e294fe4aceb24919e9e2bddf65
I20250625 13:51:56.648386 1165 ts_tablet_manager.cc:1939] T 7fe053e294fe4aceb24919e9e2bddf65 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.649255 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 223) took 1156 ms. Trace:
I20250625 13:51:56.649443 1165 rpcz_store.cc:276] 0625 13:51:55.492783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.492835 (+ 52us) service_pool.cc:225] Handling call
0625 13:51:56.611600 (+1118765us) ts_tablet_manager.cc:1145] Deleting tablet 7fe053e294fe4aceb24919e9e2bddf65
0625 13:51:56.611611 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.634909 (+ 23298us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.647259 (+ 12350us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.649217 (+ 1958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118534}
I20250625 13:51:56.650069 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 224) took 1080 ms. Trace:
I20250625 13:51:56.650229 1165 rpcz_store.cc:276] 0625 13:51:55.569393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.569964 (+ 571us) service_pool.cc:225] Handling call
0625 13:51:56.649928 (+1079964us) ts_tablet_manager.cc:1145] Deleting tablet 9a31f10c8ca745b7ad635c622a265450
0625 13:51:56.649937 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.650038 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076492}
I20250625 13:51:56.650812 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 225) took 1058 ms. Trace:
I20250625 13:51:56.651135 1165 rpcz_store.cc:276] 0625 13:51:55.592149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.592856 (+ 707us) service_pool.cc:225] Handling call
0625 13:51:56.650672 (+1057816us) ts_tablet_manager.cc:1145] Deleting tablet 8b7062c34fd143bf86b743055a27db8c
0625 13:51:56.650683 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.650782 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057481}
I20250625 13:51:56.651691 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 0723e91650174f5badd74f7756c5a54e (table default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
I20250625 13:51:56.653723 1165 tablet_replica.cc:331] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.653990 1165 raft_consensus.cc:2241] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:56.654284 1165 pending_rounds.cc:70] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:56.654170 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 (DEFAULT_TABLE table=default.Table_de6ae2243e604e0a8094e43c5b76ca27 [id=6de15b35e99646eb8eba9c11eea579c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.654457 1165 pending_rounds.cc:77] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520578964037632 op_type: NO_OP noop_request { }
I20250625 13:51:56.654655 1165 raft_consensus.cc:2887] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:56.654734 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c5fc63828d9408b9cf774a966f43cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.654824 1165 raft_consensus.cc:2270] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.655776 1165 ts_tablet_manager.cc:1905] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:56.656062 581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250625 13:51:56.656455 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d4f19ddc51324cc18242c6081f3f7de5 (table default.Table_f2cf2aab62814c7389fa4b446b696dd1 [id=b8285b64bbee49f9be8741b252d03363]). Expected version 1 got 0
I20250625 13:51:56.656682 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 3 got 0
I20250625 13:51:56.656869 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]). Expected version 1 got 0
I20250625 13:51:56.661451 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 with delete_type TABLET_DATA_DELETED (replaced by tablet 32143e2497ee4d1b8c2e7cbfc6160ef4) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.662493 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 (DEFAULT_TABLE table=default.Table_c7891a214410484c907ee024a3f40331 [id=21a2a642fff348518e79d4e4db1f32ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.663637 745 tablet_service.cc:1468] Processing CreateTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 (DEFAULT_TABLE table=default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae [id=c4babc967a48425587250196c8d9b4ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.664078 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d01af6b320489fb99c467c11ed1dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.663990 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 74884bf313724d678d3b5f5869f09f97 (DEFAULT_TABLE table=default.Table_25d9c73fdf1b43b787345769f54915d6 [id=79ace8a081c447a5b4c44b5911a906c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.664628 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 490c8902efeb46c99543747a8787c1c7 (DEFAULT_TABLE table=default.Table_dacf434b035e41c293c9fb0057989436 [id=a0e4a71b13d94a69affc3cc9e0cb5066]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.664669 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 5e280bd329764924a6fd42c591888ed0 (DEFAULT_TABLE table=default.Table_2211bebab65640d09cfecd34b8eaaa40 [id=3d206a6b57d34bf599390f47c855c11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.664533 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8063122ea7b24b6790f5c3cc884e09be (DEFAULT_TABLE table=default.Table_9e22da664adb404cad353db7745d0110 [id=75f0bde5d60641f8afe279bd82d16ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.666102 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 7fe053e294fe4aceb24919e9e2bddf65 (table default.Table_d9dd2f12f8bb4f6cba435c892dac832e [id=98f8967275434f48a948def71ae1d693]) successfully deleted
I20250625 13:51:56.666172 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8063122ea7b24b6790f5c3cc884e09be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.666354 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9a31f10c8ca745b7ad635c622a265450 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a31f10c8ca745b7ad635c622a265450 already in progress: creating tablet
W20250625 13:51:56.666548 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8b7062c34fd143bf86b743055a27db8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b7062c34fd143bf86b743055a27db8c already in progress: creating tablet
W20250625 13:51:56.666718 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8fdca99b734448118e0c42227c78e58f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fdca99b734448118e0c42227c78e58f already in progress: creating tablet
W20250625 13:51:56.666945 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8e29e7b55570483b80c6e7a1f227fa7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e29e7b55570483b80c6e7a1f227fa7b already in progress: creating tablet
W20250625 13:51:56.667104 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 637847cac1724bfeb629c4d4013396ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 637847cac1724bfeb629c4d4013396ca already in progress: creating tablet
W20250625 13:51:56.667279 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 54fa042947da4e1b89018640b46b271a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54fa042947da4e1b89018640b46b271a already in progress: creating tablet
W20250625 13:51:56.667532 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 58aa3c21191e4bea9c5395d6e32b8717 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58aa3c21191e4bea9c5395d6e32b8717 already in progress: creating tablet
I20250625 13:51:56.667683 753 tablet_service.cc:1468] Processing CreateTablet for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (DEFAULT_TABLE table=default.Table_c3f75eb1454e43e2a012dad91ae07d37 [id=bd5767cb7ca1457a9f41dbad6535b38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.667924 743 tablet_service.cc:1468] Processing CreateTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 (DEFAULT_TABLE table=default.Table_3df36037c9584e4c94d313cdf1402fdd [id=d1dc28e680ae47fe8b0bce895330e353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.668017 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.668260 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70959ec681524518a7ec2c9ed4016ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.668644 749 tablet_service.cc:1468] Processing CreateTablet for tablet 723246e1e6824faf9f9d39c3ea941fe4 (DEFAULT_TABLE table=default.Table_42f697776fd04dc3ab9be41c229289fe [id=1178d7027158455bb95716dd9e102ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.668962 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723246e1e6824faf9f9d39c3ea941fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.669876 750 tablet_service.cc:1468] Processing CreateTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 (DEFAULT_TABLE table=default.Table_068bedd508da462681c0138ddd8964e3 [id=3a4d573b5eb84c639bf289ec308eb50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.670332 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b8c8aa8036454298bf1e81bab91c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.672468 752 tablet_service.cc:1468] Processing CreateTablet for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (DEFAULT_TABLE table=default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.675542 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0568e1a3b89f4380abb0d3897817ccef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.676168 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0568e1a3b89f4380abb0d3897817ccef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.678577 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 3 got 0
I20250625 13:51:56.679132 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 991bb428def545fd979e9f1a80ce4d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.679952 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.680183 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.681223 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 067272869c5547648c90f83a34a036aa consensus state has no leader
I20250625 13:51:56.683385 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 36da1da539e24f55978d516873cb0b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.683589 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 389d81329b8d430a9098b0630612fdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.685470 740 tablet_service.cc:1515] Processing DeleteTablet for tablet b7641869a0bb463698244ff445d87a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.691474 1166 tablet_bootstrap.cc:492] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.692550 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3e7baa9cc87143b0833a6bc32a193408"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.693185 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e7baa9cc87143b0833a6bc32a193408 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.694535 1166 ts_tablet_manager.cc:1397] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.007s
I20250625 13:51:56.694698 892 tablet_service.cc:1468] Processing CreateTablet for tablet 490c8902efeb46c99543747a8787c1c7 (DEFAULT_TABLE table=default.Table_dacf434b035e41c293c9fb0057989436 [id=a0e4a71b13d94a69affc3cc9e0cb5066]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.695034 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a9b04cfcee6646909171bdaf821ff85e" } modify_external_catalogs: true
I20250625 13:51:56.695195 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a9b04cfcee6646909171bdaf821ff85e" } modify_external_catalogs: true
I20250625 13:51:56.695277 888 tablet_service.cc:1468] Processing CreateTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 (DEFAULT_TABLE table=default.Table_de6ae2243e604e0a8094e43c5b76ca27 [id=6de15b35e99646eb8eba9c11eea579c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.696017 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 with delete_type TABLET_DATA_DELETED (replaced by tablet 32143e2497ee4d1b8c2e7cbfc6160ef4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.696296 887 tablet_service.cc:1468] Processing CreateTablet for tablet 024f6cb363594420b0d5480fa79d156b (DEFAULT_TABLE table=default.Table_0892d2b459334a8bac680a69f2be033e [id=3eae0d10bcb247e0a318d6ac81e8c5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.699852 1302 raft_consensus.cc:491] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.701503 1166 raft_consensus.cc:357] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.701773 1165 ts_tablet_manager.cc:1918] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:56.701913 1165 log.cc:1199] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/1a7b72c2c464421eacfcf64008c3aea8
I20250625 13:51:56.703410 1166 raft_consensus.cc:383] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.703516 1166 raft_consensus.cc:738] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.703931 1166 consensus_queue.cc:260] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.704387 1166 ts_tablet_manager.cc:1428] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 13:51:56.705082 1166 tablet_bootstrap.cc:492] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.709203 1166 tablet_bootstrap.cc:654] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.710516 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 with delete_type TABLET_DATA_DELETED (replaced by tablet f00e72e103f443e886f0df690cff4b6f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.714052 1165 ts_tablet_manager.cc:1939] T 1a7b72c2c464421eacfcf64008c3aea8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:56.715193 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 1a7b72c2c464421eacfcf64008c3aea8 (table default.Table_3599147b4b6b4aa3bcaa6de4475ae800 [id=cd3d67e4b72d4d919344482f7b10ed10]) successfully deleted
I20250625 13:51:56.716806 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 20c12f55ed5b4425aea1bf45cdc0658e (DEFAULT_TABLE table=default.Table_6b0fc9a5299d475aa3ac845d06953e17 [id=06980d9c71814771ae6ccb685b4e54c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.717049 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 (DEFAULT_TABLE table=default.Table_3462a32d0970423c8538a79dae984471 [id=60ccb9d481c14fd68670e5701ce76b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.700160 1302 raft_consensus.cc:513] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.718466 1302 leader_election.cc:290] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:56.719229 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c0a3ea18bab944749d45da9b3139f8bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0a3ea18bab944749d45da9b3139f8bc already in progress: creating tablet
I20250625 13:51:56.719625 1165 tablet_replica.cc:331] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:56.721917 1165 raft_consensus.cc:2241] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:56.722295 1165 raft_consensus.cc:2270] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:56.722501 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cf73b4ab99c466abcfedf1b200e743b consensus state has no leader
I20250625 13:51:56.722936 1165 ts_tablet_manager.cc:1905] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:56.730177 1167 ts_tablet_manager.cc:1918] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:56.730541 1167 log.cc:1199] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/99090fee41d44946b6243b1b4bff3635
I20250625 13:51:56.731611 1167 ts_tablet_manager.cc:1939] T 99090fee41d44946b6243b1b4bff3635 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:56.731809 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc
I20250625 13:51:56.732738 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 46316eb725bc47e695af476c0b185504 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:56.732807 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 224) took 1056 ms. Trace:
I20250625 13:51:56.733048 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.733129 1167 rpcz_store.cc:276] 0625 13:51:55.676067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.676482 (+ 415us) service_pool.cc:225] Handling call
0625 13:51:56.194333 (+517851us) ts_tablet_manager.cc:1145] Deleting tablet 99090fee41d44946b6243b1b4bff3635
0625 13:51:56.194345 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.404187 (+209842us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.709503 (+305316us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.732726 (+ 23223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":517565}
I20250625 13:51:56.734244 1167 tablet_replica.cc:331] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:56.734664 1167 raft_consensus.cc:2241] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:56.735158 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a29708fb688a4a65b84ae2564a040b4c (DEFAULT_TABLE table=default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.736002 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6c5fc63828d9408b9cf774a966f43cf2
I20250625 13:51:56.736541 1166 tablet_bootstrap.cc:492] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:56.736711 1166 ts_tablet_manager.cc:1397] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250625 13:51:56.737644 1166 raft_consensus.cc:357] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:56.737903 1166 raft_consensus.cc:383] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.737996 1166 raft_consensus.cc:738] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.738242 1166 consensus_queue.cc:260] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:56.738245 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.738859 1166 ts_tablet_manager.cc:1428] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:56.740146 1166 tablet_bootstrap.cc:492] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:56.743605 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef with delete_type TABLET_DATA_DELETED (replaced by tablet 88f72c555d9842f28ca4bf310da9c53c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.747875 1166 tablet_bootstrap.cc:654] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:56.760882 882 tablet_service.cc:1468] Processing CreateTablet for tablet 042359eacaa2451abc0fa6ed447a2905 (DEFAULT_TABLE table=default.Table_f0caf6cb5cc84df9b75703d4a12eb505 [id=426662edb72f4d1aa3e0aa1ee934e62d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.762317 1171 ts_tablet_manager.cc:1428] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.077s user 0.001s sys 0.003s
I20250625 13:51:56.762998 894 tablet_service.cc:1468] Processing CreateTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 (DEFAULT_TABLE table=default.Table_c7891a214410484c907ee024a3f40331 [id=21a2a642fff348518e79d4e4db1f32ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.766819 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c7d9dab52e4c4c3bbd399e6dbc197775" } modify_external_catalogs: true
I20250625 13:51:56.767055 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c7d9dab52e4c4c3bbd399e6dbc197775" } modify_external_catalogs: true
I20250625 13:51:56.767920 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 46316eb725bc47e695af476c0b185504
I20250625 13:51:56.768409 588 catalog_manager.cc:6652] ReplaceTablet: tablet 46316eb725bc47e695af476c0b185504 deleted and replaced by tablet b7d888e628a94f939ecf2571e60b1948
I20250625 13:51:56.735591 1167 raft_consensus.cc:2270] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:56.746711 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba1338d0562418eadabdb8dbcaedee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.769127 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 46316eb725bc47e695af476c0b185504 with delete_type TABLET_DATA_DELETED (replaced by tablet b7d888e628a94f939ecf2571e60b1948) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.769152 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8ba86fb74c0940fcaec4f753e8139f4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.769454 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c7891a214410484c907ee024a3f40331" } modify_external_catalogs: true
I20250625 13:51:56.769613 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c7891a214410484c907ee024a3f40331" } modify_external_catalogs: true
W20250625 13:51:56.769704 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ba86fb74c0940fcaec4f753e8139f4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.770843 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_dc86aa488ba944c689ae858bc3800718"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.771299 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc86aa488ba944c689ae858bc3800718 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.747278 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet bb1c7a85da844d34be805843bc60d350 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.771934 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490c8902efeb46c99543747a8787c1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.772138 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_89e73d4233d14f7ab1805e3351155943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.772449 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9514bbbda0ba4329a18b0a42e635c6e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.772593 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89e73d4233d14f7ab1805e3351155943 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.772616 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_13c3269d65ff41bc87efff42b2dce296" } modify_external_catalogs: true
I20250625 13:51:56.773319 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_13c3269d65ff41bc87efff42b2dce296" } modify_external_catalogs: true
I20250625 13:51:56.775200 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c12f55ed5b4425aea1bf45cdc0658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.779551 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74884bf313724d678d3b5f5869f09f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.781740 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet df1ed922750f45eab02d1fc0c26af75e
I20250625 13:51:56.782948 750 tablet_service.cc:1515] Processing DeleteTablet for tablet df1ed922750f45eab02d1fc0c26af75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.786620 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 29be8c0350584de382985e66556bdcca
I20250625 13:51:56.787366 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7ba1338d0562418eadabdb8dbcaedee2
I20250625 13:51:56.788511 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.788776 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 29be8c0350584de382985e66556bdcca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.751724 1465 raft_consensus.cc:3058] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.791126 750 tablet_service.cc:1468] Processing CreateTablet for tablet b54183cf36b049c7bed30deb1604c59a (DEFAULT_TABLE table=default.Table_fdfdf918b84c4bc7874ae0a62279c7c7 [id=2bf141a440794be1919feb01f47d5009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.754066 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b0037d09796455cbaaadfaa9c010673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.754189 1174 raft_consensus.cc:2747] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:56.791556 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54183cf36b049c7bed30deb1604c59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.755520 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 643928ca4fc94ce58431a0525f2abb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:56.755723 1501 raft_consensus.cc:491] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.792042 1501 raft_consensus.cc:513] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.792917 1501 leader_election.cc:290] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.794090 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 04479dfb34584f0891c604ef72bf93b5 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:56.794152 985 leader_election.cc:343] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1a7b72c2c464421eacfcf64008c3aea8
I20250625 13:51:56.795725 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40ca12b3cc9a4632912f3508933960e2" } new_table_name: "default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3" modify_external_catalogs: true
I20250625 13:51:56.796551 1465 raft_consensus.cc:2747] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:56.802187 1167 ts_tablet_manager.cc:1905] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:56.806918 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79beb23eebf54fd9951ea5a33fcfb0ae consensus state has no leader
I20250625 13:51:56.807565 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4a201c5623af4b8a95f24f43319311f0" } new_table_name: "default.Table_e0ded9bf53314d66a9363d735e000f5c" modify_external_catalogs: true
I20250625 13:51:56.812873 755 tablet_service.cc:1468] Processing CreateTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 (DEFAULT_TABLE table=default.Table_3ac6f01366104d75a5bf908bc40c3f4f [id=8a9015750b254a9ea6a48a9510aeddf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.813441 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e4bd4aaab143ffa65a668a952ae345. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.760119 1170 tablet_bootstrap.cc:492] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:56.817579 1170 ts_tablet_manager.cc:1397] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.528s user 0.004s sys 0.022s
I20250625 13:51:56.818408 1170 raft_consensus.cc:357] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.818619 1170 raft_consensus.cc:383] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:56.818501 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_06d8369e7a09451eba488006b53da953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.818692 1170 raft_consensus.cc:738] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:56.818899 1170 consensus_queue.cc:260] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:56.819205 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06d8369e7a09451eba488006b53da953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:56.820219 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dad4ea8cf44944b08f53428c95809daf consensus state has no leader
I20250625 13:51:56.822783 896 tablet_service.cc:1468] Processing CreateTablet for tablet 5e280bd329764924a6fd42c591888ed0 (DEFAULT_TABLE table=default.Table_2211bebab65640d09cfecd34b8eaaa40 [id=3d206a6b57d34bf599390f47c855c11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.823282 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7641869a0bb463698244ff445d87a1f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:56.823226 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 11711546f0e14f9a9eabf003f49a69b7 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:56.824126 711 leader_election.cc:343] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.825433 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e280bd329764924a6fd42c591888ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.826622 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ac6eba07f2c94c75a0be2404ab15f0d9" } new_table_name: "default.Table_dbca3ace8835418e8db2cbc1845d9cfb" modify_external_catalogs: true
I20250625 13:51:56.826819 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13620f6ca65342b4ad7f1ba4b437bde3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:56.827533 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7641869a0bb463698244ff445d87a1f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:56.828402 711 leader_election.cc:343] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:56.828981 711 leader_election.cc:343] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.829099 711 leader_election.cc:304] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:56.829505 1302 raft_consensus.cc:2747] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:56.830472 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 424709us
I20250625 13:51:56.831887 1170 ts_tablet_manager.cc:1428] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250625 13:51:56.833366 1170 tablet_bootstrap.cc:492] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:56.835916 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_eaf03b2c45a842c392e299f6fe84dc05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.836570 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaf03b2c45a842c392e299f6fe84dc05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.837634 1167 ts_tablet_manager.cc:1918] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:56.839489 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 11711546f0e14f9a9eabf003f49a69b7
I20250625 13:51:56.840083 588 catalog_manager.cc:6652] ReplaceTablet: tablet 11711546f0e14f9a9eabf003f49a69b7 deleted and replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:51:56.842353 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 with delete_type TABLET_DATA_DELETED (replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:56.844296 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ba63184ca2b4badafcf08309ef75164 consensus state has no leader
I20250625 13:51:56.846745 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3859f399a7b84de99658380bd6c24ff5 (DEFAULT_TABLE table=default.Table_8d37b99f76eb49c8af370ba05e4015aa [id=27acc786d57a4c7ab531242dd52f17ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.847357 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3859f399a7b84de99658380bd6c24ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.850524 893 tablet_service.cc:1468] Processing CreateTablet for tablet 20c12f55ed5b4425aea1bf45cdc0658e (DEFAULT_TABLE table=default.Table_6b0fc9a5299d475aa3ac845d06953e17 [id=06980d9c71814771ae6ccb685b4e54c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.851545 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 643928ca4fc94ce58431a0525f2abb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.851783 1302 raft_consensus.cc:491] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.851984 1302 raft_consensus.cc:513] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.852298 883 tablet_service.cc:1468] Processing CreateTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 (DEFAULT_TABLE table=default.Table_3462a32d0970423c8538a79dae984471 [id=60ccb9d481c14fd68670e5701ce76b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.852841 1302 leader_election.cc:290] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.853297 890 tablet_service.cc:1468] Processing CreateTablet for tablet a29708fb688a4a65b84ae2564a040b4c (DEFAULT_TABLE table=default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.854681 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 991bb428def545fd979e9f1a80ce4d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.855198 884 tablet_service.cc:1468] Processing CreateTablet for tablet 74884bf313724d678d3b5f5869f09f97 (DEFAULT_TABLE table=default.Table_25d9c73fdf1b43b787345769f54915d6 [id=79ace8a081c447a5b4c44b5911a906c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.856005 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b0037d09796455cbaaadfaa9c010673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.859218 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02f2a2b595bc43309a29b3f1bc5d3766" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:56.859879 711 leader_election.cc:343] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:56.864653 753 tablet_service.cc:1468] Processing CreateTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e (DEFAULT_TABLE table=default.Table_3be2a3bc86fd4fd993dbfa89af6b7b97 [id=317ce1c5a5774d6ca8d00ed976bf8615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.866889 749 tablet_service.cc:1468] Processing CreateTablet for tablet dcf7fab53cae49908d2952cdaa07943a (DEFAULT_TABLE table=default.Table_5513b6e4edc14bdea44f9e8562b4d70e [id=aa4ef3fe7bbb408396878d79f7caffb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.867595 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe4046742b34f1ab0b1610b5d6ee01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.876320 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c5fc63828d9408b9cf774a966f43cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.879350 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf7fab53cae49908d2952cdaa07943a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.882115 849 consensus_peers.cc:487] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:51:56.882284 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 52308.3us
I20250625 13:51:56.882602 878 tablet_service.cc:1515] Processing DeleteTablet for tablet bb1c7a85da844d34be805843bc60d350 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:56.885509 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a7b72c2c464421eacfcf64008c3aea8" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:56.885892 914 raft_consensus.cc:2391] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:51:56.887956 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74884bf313724d678d3b5f5869f09f97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.889807 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_14940f9e4a2645aab4d9b490b94171b3" } new_table_name: "default.Table_40de6f83d3af49f1a0bc517c4d0d875e" modify_external_catalogs: true
W20250625 13:51:56.896710 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9bb7ebc40a14ecaa0c692a8e1117db3 consensus state has no leader
I20250625 13:51:56.899878 1307 raft_consensus.cc:491] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.904047 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9514bbbda0ba4329a18b0a42e635c6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.904505 1302 raft_consensus.cc:491] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:56.904736 1302 raft_consensus.cc:513] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.905524 1302 leader_election.cc:290] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.906672 1171 tablet_bootstrap.cc:492] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:56.910549 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:56.910977 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 042359eacaa2451abc0fa6ed447a2905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.912957 1307 raft_consensus.cc:513] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:56.917042 581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:33015, user_credentials={real_user=slave}} blocked reactor thread for 260900us
I20250625 13:51:56.918675 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba1338d0562418eadabdb8dbcaedee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.927134 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8d37b99f76eb49c8af370ba05e4015aa" } new_table_name: "default.Table_cf29f4b4bbb94091affd4440a4b8447c" modify_external_catalogs: true
I20250625 13:51:56.928258 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:56.928512 777 raft_consensus.cc:2466] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:56.929425 753 tablet_service.cc:1468] Processing CreateTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 (DEFAULT_TABLE table=default.Table_3e7baa9cc87143b0833a6bc32a193408 [id=d043c74a6cb94efa89ce819d2ef35545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.929941 743 tablet_service.cc:1468] Processing CreateTablet for tablet dfaba56442c0498da95e6fad126ab86d (DEFAULT_TABLE table=default.Table_eaf03b2c45a842c392e299f6fe84dc05 [id=7270b5aa9a0148178683a229db5b4b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.931088 755 tablet_service.cc:1468] Processing CreateTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 (DEFAULT_TABLE table=default.Table_dc86aa488ba944c689ae858bc3800718 [id=028f5254a2c045929bdb41c0b273875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.931607 750 tablet_service.cc:1468] Processing CreateTablet for tablet 9a2007e51d424053bc2836dedf46add2 (DEFAULT_TABLE table=default.Table_0568e1a3b89f4380abb0d3897817ccef [id=fa06a5b0571b4a3a8b9b68f1f05fc3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.933471 749 tablet_service.cc:1468] Processing CreateTablet for tablet 569f6cba69394f2fa005686056f19929 (DEFAULT_TABLE table=default.Table_06d8369e7a09451eba488006b53da953 [id=421fdfc88bfe40349eae76010c55e7ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.933933 745 tablet_service.cc:1468] Processing CreateTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 (DEFAULT_TABLE table=default.Table_89e73d4233d14f7ab1805e3351155943 [id=fe513f20709d46d3832d046a1544c47f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.934338 752 tablet_service.cc:1468] Processing CreateTablet for tablet dec08d20a09444f5b9a102a8712e36c7 (DEFAULT_TABLE table=default.Table_8ba86fb74c0940fcaec4f753e8139f4a [id=d29821c37b644033be9a32f6adfb9710]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.934791 740 tablet_service.cc:1468] Processing CreateTablet for tablet b7d888e628a94f939ecf2571e60b1948 (DEFAULT_TABLE table=default.Table_7a2ca389ecd148b9be0e90c130fe6431 [id=26d0bdba8be543d89d5337c86bbf6e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.935727 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e280bd329764924a6fd42c591888ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.936152 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3c72767e52f24da8b58c00aa584d15c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.936864 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c72767e52f24da8b58c00aa584d15c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.938242 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_81732e5a747642cd8d61556c2ed6f6c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:56.938691 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81732e5a747642cd8d61556c2ed6f6c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.937713 914 raft_consensus.cc:2466] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:56.939967 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_06ce856b6c374443991c4f9ce9a60eb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.940140 851 leader_election.cc:304] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:51:56.940397 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ce856b6c374443991c4f9ce9a60eb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.940599 889 tablet_service.cc:1468] Processing CreateTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 (DEFAULT_TABLE table=default.Table_068bedd508da462681c0138ddd8964e3 [id=3a4d573b5eb84c639bf289ec308eb50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.940992 885 tablet_service.cc:1468] Processing CreateTablet for tablet 8063122ea7b24b6790f5c3cc884e09be (DEFAULT_TABLE table=default.Table_9e22da664adb404cad353db7745d0110 [id=75f0bde5d60641f8afe279bd82d16ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.943007 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40de6f83d3af49f1a0bc517c4d0d875e" } modify_external_catalogs: true
I20250625 13:51:56.943221 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ba435656502a49659e22a61038cad9eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.943538 711 leader_election.cc:304] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:51:56.943737 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba435656502a49659e22a61038cad9eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.943998 1302 raft_consensus.cc:2802] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:56.944046 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3462a32d0970423c8538a79dae984471" } modify_external_catalogs: true
I20250625 13:51:56.944106 1302 raft_consensus.cc:491] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:56.943032 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_65410f67a07044e9a0b308de1bd53c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:56.944223 1302 raft_consensus.cc:3058] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:56.944583 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65410f67a07044e9a0b308de1bd53c34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:56.943199 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40de6f83d3af49f1a0bc517c4d0d875e" } modify_external_catalogs: true
I20250625 13:51:56.946846 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3462a32d0970423c8538a79dae984471" } modify_external_catalogs: true
I20250625 13:51:56.948100 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c12f55ed5b4425aea1bf45cdc0658e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.949553 1302 raft_consensus.cc:513] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:56.952040 1302 leader_election.cc:290] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:56.952464 741 tablet_service.cc:1468] Processing CreateTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 (DEFAULT_TABLE table=default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.953169 758 tablet_service.cc:1468] Processing CreateTablet for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (DEFAULT_TABLE table=default.Table_25705d30e7034c57be006958705ab688 [id=3977ee3968d246b7a8aed1fafd1822b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.953737 757 tablet_service.cc:1468] Processing CreateTablet for tablet a30a35e3639a43c5b078169060824723 (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.954286 747 tablet_service.cc:1468] Processing CreateTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 (DEFAULT_TABLE table=default.Table_531f237d57114b2483ae3cf8d4b94659 [id=a525a4b043274f8ea88049ed7d1cf652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.955219 1307 leader_election.cc:290] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:56.955905 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b8c8aa8036454298bf1e81bab91c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.961370 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8063122ea7b24b6790f5c3cc884e09be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:56.962221 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3859f399a7b84de99658380bd6c24ff5 consensus state has no leader
I20250625 13:51:56.956794 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:56.964522 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29708fb688a4a65b84ae2564a040b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.957168 1424 raft_consensus.cc:2802] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:56.967938 1424 raft_consensus.cc:491] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:56.968091 1424 raft_consensus.cc:3058] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:56.969610 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 024f6cb363594420b0d5480fa79d156b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.970289 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d9bb7ebc40a14ecaa0c692a8e1117db3
I20250625 13:51:56.974133 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9514bbbda0ba4329a18b0a42e635c6e5
I20250625 13:51:56.975749 756 tablet_service.cc:1515] Processing DeleteTablet for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.976015 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:56.979746 886 tablet_service.cc:1468] Processing CreateTablet for tablet 723246e1e6824faf9f9d39c3ea941fe4 (DEFAULT_TABLE table=default.Table_42f697776fd04dc3ab9be41c229289fe [id=1178d7027158455bb95716dd9e102ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.980499 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723246e1e6824faf9f9d39c3ea941fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.981968 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490c8902efeb46c99543747a8787c1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.984459 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8c4176baf42b43658ec245c90d86cb98" } modify_external_catalogs: true
I20250625 13:51:56.984673 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8c4176baf42b43658ec245c90d86cb98" } modify_external_catalogs: true
I20250625 13:51:56.985497 895 tablet_service.cc:1468] Processing CreateTablet for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (DEFAULT_TABLE table=default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:56.986107 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.988226 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ef50a3624b4c79acc803ff0e2ed364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.988804 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfaba56442c0498da95e6fad126ab86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.989343 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2f8d0d9a714a28ba6fb9642d461de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.989861 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2007e51d424053bc2836dedf46add2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.990423 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569f6cba69394f2fa005686056f19929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.990932 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de9656958164dc4ae2fb0078b38f6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.991462 1167 log.cc:1199] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/32221a413957450fa59ffcfbc91c51b9
I20250625 13:51:56.991534 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dec08d20a09444f5b9a102a8712e36c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.992038 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7d888e628a94f939ecf2571e60b1948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.992512 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c6a325d40c4f7ba2f342e39b9437e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.993048 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30a35e3639a43c5b078169060824723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.993546 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69672579a83c41d09ee7c78a6fa7d746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.994148 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c605d5b64c164c1594b0a40cfa2c4cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:56.995127 1167 ts_tablet_manager.cc:1939] T 32221a413957450fa59ffcfbc91c51b9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:56.998024 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 222) took 1320 ms. Trace:
I20250625 13:51:56.998145 1167 rpcz_store.cc:276] 0625 13:51:55.675754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.676522 (+ 768us) service_pool.cc:225] Handling call
0625 13:51:56.734111 (+1057589us) ts_tablet_manager.cc:1145] Deleting tablet 32221a413957450fa59ffcfbc91c51b9
0625 13:51:56.734122 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.805613 (+ 71491us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.837495 (+ 31882us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:56.996137 (+158642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1056440}
I20250625 13:51:56.998759 983 leader_election.cc:304] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:56.999432 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 228) took 1285 ms. Trace:
I20250625 13:51:56.999554 1167 rpcz_store.cc:276] 0625 13:51:55.713249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.713801 (+ 552us) service_pool.cc:225] Handling call
0625 13:51:56.998526 (+1284725us) ts_tablet_manager.cc:1145] Deleting tablet 8fdca99b734448118e0c42227c78e58f
0625 13:51:56.998540 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.998708 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1284448}
I20250625 13:51:56.991488 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29708fb688a4a65b84ae2564a040b4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.000646 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 113377us
I20250625 13:51:57.003060 1170 tablet_bootstrap.cc:654] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:57.005291 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 (DEFAULT_TABLE table=default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae [id=c4babc967a48425587250196c8d9b4ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.006858 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 (DEFAULT_TABLE table=default.Table_3df36037c9584e4c94d313cdf1402fdd [id=d1dc28e680ae47fe8b0bce895330e353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.010550 1465 raft_consensus.cc:3058] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.007582 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:57.015622 849 leader_election.cc:343] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.021934 1465 raft_consensus.cc:2747] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.022163 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 230) took 1293 ms. Trace:
I20250625 13:51:57.022279 1167 rpcz_store.cc:276] 0625 13:51:55.713446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.715255 (+ 1809us) service_pool.cc:225] Handling call
0625 13:51:57.004581 (+1289326us) ts_tablet_manager.cc:1145] Deleting tablet 2bbd509e3dfb45d98314c6afe37677f6
0625 13:51:57.004595 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.006423 (+ 1828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287174}
I20250625 13:51:57.022756 1167 tablet_replica.cc:331] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:57.022992 1167 raft_consensus.cc:2241] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:57.023209 1167 raft_consensus.cc:2270] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:57.023950 1167 ts_tablet_manager.cc:1905] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:57.024139 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (DEFAULT_TABLE table=default.Table_c3f75eb1454e43e2a012dad91ae07d37 [id=bd5767cb7ca1457a9f41dbad6535b38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.024924 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 (DEFAULT_TABLE table=default.Table_068bedd508da462681c0138ddd8964e3 [id=3a4d573b5eb84c639bf289ec308eb50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.027514 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:57.028381 711 leader_election.cc:343] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.030450 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 8063122ea7b24b6790f5c3cc884e09be (DEFAULT_TABLE table=default.Table_9e22da664adb404cad353db7745d0110 [id=75f0bde5d60641f8afe279bd82d16ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.031002 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:57.036890 1302 raft_consensus.cc:491] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.037204 1302 raft_consensus.cc:513] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.038203 1302 leader_election.cc:290] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.039264 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "389d81329b8d430a9098b0630612fdcb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:57.040370 711 leader_election.cc:343] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:57.040693 711 leader_election.cc:343] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.043355 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0f521fe0405e4837b9b41acdc5b12c51" } new_table_name: "default.Table_55953048150049c490cd5b9346184cd0" modify_external_catalogs: true
I20250625 13:51:57.044456 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.044720 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.045017 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (DEFAULT_TABLE table=default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.045524 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae" } modify_external_catalogs: true
I20250625 13:51:57.045699 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae" } modify_external_catalogs: true
I20250625 13:51:57.045768 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet df1ed922750f45eab02d1fc0c26af75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.046216 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 46316eb725bc47e695af476c0b185504 with delete_type TABLET_DATA_DELETED (replaced by tablet b7d888e628a94f939ecf2571e60b1948) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.046604 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 723246e1e6824faf9f9d39c3ea941fe4 (DEFAULT_TABLE table=default.Table_42f697776fd04dc3ab9be41c229289fe [id=1178d7027158455bb95716dd9e102ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.046690 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8b7b90e3ff0f4b81bef04d3952bae1ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.046998 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "389d81329b8d430a9098b0630612fdcb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:57.047375 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b7b90e3ff0f4b81bef04d3952bae1ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.047923 1302 raft_consensus.cc:491] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.048115 1302 raft_consensus.cc:513] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.048868 1302 leader_election.cc:290] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.049288 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_866deeb74a8743f0a579513bf580c9e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.049914 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_866deeb74a8743f0a579513bf580c9e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:57.050304 711 leader_election.cc:343] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.050485 711 leader_election.cc:304] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:57.051458 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87327f74beb845768bbfd5ae6d2bcf88" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:57.051710 915 raft_consensus.cc:2466] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:57.052531 1302 raft_consensus.cc:2747] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.052459 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e7bc36f60d844f0d9d27755d6f58f63b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.053011 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7bc36f60d844f0d9d27755d6f58f63b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.053138 711 leader_election.cc:304] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:57.053601 1302 raft_consensus.cc:2802] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:57.053707 1302 raft_consensus.cc:491] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:57.053826 1302 raft_consensus.cc:3058] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.054352 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328" } modify_external_catalogs: true
I20250625 13:51:57.054495 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328" } modify_external_catalogs: true
I20250625 13:51:57.056672 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_61827a08a660423a91ddac5834f61382"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.057222 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61827a08a660423a91ddac5834f61382 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.057668 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 with delete_type TABLET_DATA_DELETED (replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.058622 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d46618ecfae24fe2bdd9fc2feea12ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.059197 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fe053e294fe4aceb24919e9e2bddf65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:57.059589 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.059857 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b54183cf36b049c7bed30deb1604c59a (DEFAULT_TABLE table=default.Table_fdfdf918b84c4bc7874ae0a62279c7c7 [id=2bf141a440794be1919feb01f47d5009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.060002 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 29be8c0350584de382985e66556bdcca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.060091 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.061080 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 (DEFAULT_TABLE table=default.Table_3ac6f01366104d75a5bf908bc40c3f4f [id=8a9015750b254a9ea6a48a9510aeddf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.059686 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02f2a2b595bc43309a29b3f1bc5d3766" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:57.061843 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87327f74beb845768bbfd5ae6d2bcf88" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:57.062143 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1dd56390e1b04565a7850f5e960cddad" } new_table_name: "default.Table_91567a207fcd4540937c0e047ea86de5" modify_external_catalogs: true
W20250625 13:51:57.063148 711 leader_election.cc:343] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.063324 711 leader_election.cc:304] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:57.066614 1368 raft_consensus.cc:2747] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:57.067173 711 leader_election.cc:343] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.073006 1368 raft_consensus.cc:491] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.073237 1368 raft_consensus.cc:513] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.074054 1368 leader_election.cc:290] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.104843 1302 raft_consensus.cc:513] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.105813 1302 leader_election.cc:290] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.107141 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1a98dc40684954a602567cf3476879" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:57.132041 851 consensus_peers.cc:487] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf5a6951aba84f5fa8f6d76a2b7fcf2d. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:57.132495 1373 raft_consensus.cc:491] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.132609 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87327f74beb845768bbfd5ae6d2bcf88" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:57.132675 1373 raft_consensus.cc:513] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.133306 1373 leader_election.cc:290] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.138008 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.138672 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36da1da539e24f55978d516873cb0b27" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:57.139528 757 tablet_service.cc:1468] Processing CreateTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 (DEFAULT_TABLE table=default.Table_ba435656502a49659e22a61038cad9eb [id=56d2c709e389405c80e4fa20ce75d5ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.139941 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077e8c001cc646559b5268d4a48c0ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.140367 878 tablet_service.cc:1468] Processing CreateTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 (DEFAULT_TABLE table=default.Table_3df36037c9584e4c94d313cdf1402fdd [id=d1dc28e680ae47fe8b0bce895330e353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.146320 743 tablet_service.cc:1468] Processing CreateTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca (DEFAULT_TABLE table=default.Table_06ce856b6c374443991c4f9ce9a60eb1 [id=b36fcb27ef404ffeaa1532ad1063dbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.146875 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2bc4df39674dc3be0cd32c68f1ebca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.146119 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1a98dc40684954a602567cf3476879" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:57.149933 879 tablet_service.cc:1468] Processing CreateTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 (DEFAULT_TABLE table=default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae [id=c4babc967a48425587250196c8d9b4ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.150169 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87327f74beb845768bbfd5ae6d2bcf88" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:57.150410 916 raft_consensus.cc:3058] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.151105 880 tablet_service.cc:1468] Processing CreateTablet for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (DEFAULT_TABLE table=default.Table_c3f75eb1454e43e2a012dad91ae07d37 [id=bd5767cb7ca1457a9f41dbad6535b38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.151724 740 tablet_service.cc:1468] Processing CreateTablet for tablet 38770320fc6541c39e10448978f8a3b0 (DEFAULT_TABLE table=default.Table_3c72767e52f24da8b58c00aa584d15c6 [id=43487ee990e640bcbeefc7c142075998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.152204 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38770320fc6541c39e10448978f8a3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.150385 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36da1da539e24f55978d516873cb0b27" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:57.158085 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e4bd4aaab143ffa65a668a952ae345. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.167471 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54183cf36b049c7bed30deb1604c59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.168087 753 tablet_service.cc:1468] Processing CreateTablet for tablet 423ca098138c476f861a19b44554fd24 (DEFAULT_TABLE table=default.Table_81732e5a747642cd8d61556c2ed6f6c7 [id=3f116c6f6ae94239941fa04a335853ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.168578 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423ca098138c476f861a19b44554fd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.171478 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.171535 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8063122ea7b24b6790f5c3cc884e09be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.171528 1165 ts_tablet_manager.cc:1918] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:57.171756 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.171660 1165 log.cc:1199] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/7108932fa8a241819ca90674a64b285e
I20250625 13:51:57.172178 888 tablet_service.cc:1468] Processing CreateTablet for tablet b54183cf36b049c7bed30deb1604c59a (DEFAULT_TABLE table=default.Table_fdfdf918b84c4bc7874ae0a62279c7c7 [id=2bf141a440794be1919feb01f47d5009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.172478 1165 ts_tablet_manager.cc:1939] T 7108932fa8a241819ca90674a64b285e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:57.174769 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.174806 1171 tablet_bootstrap.cc:654] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:57.175859 745 tablet_service.cc:1468] Processing CreateTablet for tablet d26820eaba98401da3a8b5348be21fc6 (DEFAULT_TABLE table=default.Table_65410f67a07044e9a0b308de1bd53c34 [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.179122 1171 tablet_bootstrap.cc:492] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:57.179507 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.183426 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723246e1e6824faf9f9d39c3ea941fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.186702 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b8c8aa8036454298bf1e81bab91c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.188047 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 267) took 1087 ms. Trace:
I20250625 13:51:57.188170 1165 rpcz_store.cc:276] 0625 13:51:56.100316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.106501 (+ 6185us) service_pool.cc:225] Handling call
0625 13:51:56.719540 (+613039us) ts_tablet_manager.cc:1145] Deleting tablet 7108932fa8a241819ca90674a64b285e
0625 13:51:56.719550 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.739181 (+ 19631us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.171421 (+432240us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.187969 (+ 16548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":602603}
I20250625 13:51:57.188737 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 7108932fa8a241819ca90674a64b285e (table default.Table_194007e5294849f9a6734e39b20d4fff [id=41a6138bc94a4a2790635b4ef0757c78]) successfully deleted
I20250625 13:51:57.190124 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70959ec681524518a7ec2c9ed4016ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.192801 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 3859f399a7b84de99658380bd6c24ff5 (DEFAULT_TABLE table=default.Table_8d37b99f76eb49c8af370ba05e4015aa [id=27acc786d57a4c7ab531242dd52f17ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.193246 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3859f399a7b84de99658380bd6c24ff5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.193912 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50786a4c25274ffb8f9910c5dafe3933 consensus state has no leader
I20250625 13:51:57.194208 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d01af6b320489fb99c467c11ed1dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.194344 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f5662b7c3e9a430d87a09930ab6e8f5c" } new_table_name: "default.Table_103b9969d6414619a95c5f6da350487c" modify_external_catalogs: true
I20250625 13:51:57.196099 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d5d3601056d240ef8a91362211b21f7e" } new_table_name: "default.Table_80f2c7cdc1504002b60af89a706bec5c" modify_external_catalogs: true
I20250625 13:51:57.197551 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 1f6be7189f8f45ccb329ef5d2fd8da8d from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:57.198271 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ba435656502a49659e22a61038cad9eb" } new_table_name: "default.Table_517319d59c234fe9a5fe83530d3a00c6" modify_external_catalogs: true
I20250625 13:51:57.199352 1011 tablet_service.cc:1468] Processing CreateTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e (DEFAULT_TABLE table=default.Table_3be2a3bc86fd4fd993dbfa89af6b7b97 [id=317ce1c5a5774d6ca8d00ed976bf8615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.199819 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe4046742b34f1ab0b1610b5d6ee01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.199790 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9e86330e77034242a9aa4fbd274c3151" } modify_external_catalogs: true
I20250625 13:51:57.199966 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9e86330e77034242a9aa4fbd274c3151" } modify_external_catalogs: true
I20250625 13:51:57.201171 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 29be8c0350584de382985e66556bdcca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.201484 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_78c7bbeb91584e54bef556a0af56c2a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.202026 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78c7bbeb91584e54bef556a0af56c2a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:57.202061 1222 thread.cc:640] rpc worker (service pool kudu.server.GenericService) Time spent creating pthread: real 0.504s user 0.111s sys 0.157s
W20250625 13:51:57.202193 1222 thread.cc:606] rpc worker (service pool kudu.server.GenericService) Time spent starting thread: real 0.505s user 0.111s sys 0.157s
W20250625 13:51:57.203244 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38a23b08a03b48eaa051f670eb7dac65 consensus state has no leader
I20250625 13:51:57.203581 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_844f1dcd52aa496eb8c9c1520e7227d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.204084 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_844f1dcd52aa496eb8c9c1520e7227d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.204840 1165 tablet_replica.cc:331] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:57.205951 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26820eaba98401da3a8b5348be21fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.205955 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_12771e585c3348049cc0ab4bf2a95935"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.206368 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12771e585c3348049cc0ab4bf2a95935 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.206854 1465 raft_consensus.cc:491] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.207010 1465 raft_consensus.cc:513] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.207834 1465 leader_election.cc:290] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:57.208564 985 leader_election.cc:343] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7fe053e294fe4aceb24919e9e2bddf65
I20250625 13:51:57.209417 1165 raft_consensus.cc:2241] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:57.210522 1012 tablet_service.cc:1468] Processing CreateTablet for tablet dcf7fab53cae49908d2952cdaa07943a (DEFAULT_TABLE table=default.Table_5513b6e4edc14bdea44f9e8562b4d70e [id=aa4ef3fe7bbb408396878d79f7caffb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.210930 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf7fab53cae49908d2952cdaa07943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.211161 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70959ec681524518a7ec2c9ed4016ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.212834 1165 pending_rounds.cc:70] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:57.214233 1170 tablet_bootstrap.cc:492] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:57.214375 1170 ts_tablet_manager.cc:1397] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.381s user 0.010s sys 0.001s
I20250625 13:51:57.215476 1165 pending_rounds.cc:77] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520580148305920 op_type: NO_OP noop_request { }
I20250625 13:51:57.217085 1424 raft_consensus.cc:513] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.216107 1165 raft_consensus.cc:2887] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:57.219138 1165 raft_consensus.cc:2270] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:57.223661 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54183cf36b049c7bed30deb1604c59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.225478 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:57.225719 776 raft_consensus.cc:2391] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate affa1c3035b940e88ca143be7d35840d in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:51:57.226470 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5d7704b38144429b9c95675be84a4730
I20250625 13:51:57.226897 881 tablet_service.cc:1468] Processing CreateTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 (DEFAULT_TABLE table=default.Table_3ac6f01366104d75a5bf908bc40c3f4f [id=8a9015750b254a9ea6a48a9510aeddf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.227494 1170 raft_consensus.cc:357] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.227715 1170 raft_consensus.cc:383] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:57.227808 1170 raft_consensus.cc:738] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:57.228075 1170 consensus_queue.cc:260] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.228199 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 46316eb725bc47e695af476c0b185504 with delete_type TABLET_DATA_DELETED (replaced by tablet b7d888e628a94f939ecf2571e60b1948) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.228767 914 raft_consensus.cc:2391] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c1aa07ab6bc4b0089e46951240a37c9 in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:51:57.228865 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_32f5fbfdcc3840b8a4ff1122ff81208d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.229127 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fe053e294fe4aceb24919e9e2bddf65" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:51:57.229424 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32f5fbfdcc3840b8a4ff1122ff81208d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.230363 1170 ts_tablet_manager.cc:1428] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 13:51:57.230609 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 63d01af6b320489fb99c467c11ed1dc0
I20250625 13:51:57.231259 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f9b2ff84e19a476bb879930b39eaa142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.231724 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9b2ff84e19a476bb879930b39eaa142 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.231829 913 raft_consensus.cc:2391] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:51:57.232645 1170 tablet_bootstrap.cc:492] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:57.232982 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4fa801f7d98e4794b6ca439638f29e69
I20250625 13:51:57.233610 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3df36037c9584e4c94d313cdf1402fdd" } modify_external_catalogs: true
I20250625 13:51:57.233779 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3df36037c9584e4c94d313cdf1402fdd" } modify_external_catalogs: true
I20250625 13:51:57.236208 1165 ts_tablet_manager.cc:1905] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:57.238219 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d7704b38144429b9c95675be84a4730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.238451 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.238631 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.244453 1170 tablet_bootstrap.cc:654] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:57.248085 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be3313ec5dfb41b68662d3ff4c8c94f9 consensus state has no leader
I20250625 13:51:57.248369 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 87f5b36ddb474ed6903c14876b236ba8 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:57.249653 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c1deb52366c404cbdc08076d4bcb3d1 consensus state has no leader
W20250625 13:51:57.250027 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 077e8c001cc646559b5268d4a48c0ad6 consensus state has no leader
I20250625 13:51:57.250344 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f73f420a4d7a438eadde3d6af64f0a11
I20250625 13:51:57.255424 747 tablet_service.cc:1515] Processing DeleteTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.255924 877 tablet_service.cc:1515] Processing DeleteTablet for tablet df1ed922750f45eab02d1fc0c26af75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.257350 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.258857 1424 leader_election.cc:290] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:51:57.261147 1166 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.094s sys 0.153s
I20250625 13:51:57.263845 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d01af6b320489fb99c467c11ed1dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.266867 1168 ts_tablet_manager.cc:1918] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:57.267009 1168 log.cc:1199] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/2c1a98dc40684954a602567cf3476879
I20250625 13:51:57.267462 1168 ts_tablet_manager.cc:1939] T 2c1a98dc40684954a602567cf3476879 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:57.268038 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 70959ec681524518a7ec2c9ed4016ad1
I20250625 13:51:57.268321 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 137) took 3092 ms. Trace:
I20250625 13:51:57.269146 983 leader_election.cc:304] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:57.269585 1465 raft_consensus.cc:3058] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.270035 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:57.272735 1166 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.524s user 0.094s sys 0.153s
I20250625 13:51:57.276875 1166 tablet_bootstrap.cc:492] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:57.277052 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 87f5b36ddb474ed6903c14876b236ba8
I20250625 13:51:57.277135 1166 ts_tablet_manager.cc:1397] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.538s user 0.097s sys 0.154s
I20250625 13:51:57.277555 595 catalog_manager.cc:6652] ReplaceTablet: tablet 87f5b36ddb474ed6903c14876b236ba8 deleted and replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e
I20250625 13:51:57.278005 1166 raft_consensus.cc:357] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.278250 1166 raft_consensus.cc:383] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:57.278326 1166 raft_consensus.cc:738] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:57.278604 1166 consensus_queue.cc:260] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.279215 1166 ts_tablet_manager.cc:1428] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:51:57.268448 1168 rpcz_store.cc:276] 0625 13:51:54.176088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.177881 (+ 1793us) service_pool.cc:225] Handling call
0625 13:51:56.490043 (+2312162us) ts_tablet_manager.cc:1145] Deleting tablet 2c1a98dc40684954a602567cf3476879
0625 13:51:56.490059 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:56.731031 (+240972us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.198136 (+467105us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.268280 (+ 70144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2311934}
I20250625 13:51:57.280897 1171 ts_tablet_manager.cc:1397] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.287s user 0.006s sys 0.000s
I20250625 13:51:57.281155 877 tablet_service.cc:1468] Processing CreateTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e (DEFAULT_TABLE table=default.Table_3be2a3bc86fd4fd993dbfa89af6b7b97 [id=317ce1c5a5774d6ca8d00ed976bf8615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.283278 1171 raft_consensus.cc:357] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:57.283490 882 tablet_service.cc:1468] Processing CreateTablet for tablet dcf7fab53cae49908d2952cdaa07943a (DEFAULT_TABLE table=default.Table_5513b6e4edc14bdea44f9e8562b4d70e [id=aa4ef3fe7bbb408396878d79f7caffb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.286314 1166 tablet_bootstrap.cc:492] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:57.287421 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e281786e6e4a5d93b94b85b486655d with delete_type TABLET_DATA_DELETED (replaced by tablet be3313ec5dfb41b68662d3ff4c8c94f9) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.288328 1167 ts_tablet_manager.cc:1918] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:57.288360 1465 raft_consensus.cc:2747] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.288487 1167 log.cc:1199] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/cf5a6951aba84f5fa8f6d76a2b7fcf2d
I20250625 13:51:57.288861 1167 ts_tablet_manager.cc:1939] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:57.288911 1166 tablet_bootstrap.cc:654] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:57.289943 1170 tablet_bootstrap.cc:492] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:57.290105 1170 ts_tablet_manager.cc:1397] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.001s
I20250625 13:51:57.290156 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 with delete_type TABLET_DATA_DELETED (replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.290868 1170 raft_consensus.cc:357] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.291056 1170 raft_consensus.cc:383] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:57.291136 1170 raft_consensus.cc:738] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:57.291401 1170 consensus_queue.cc:260] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.297446 893 tablet_service.cc:1468] Processing CreateTablet for tablet 3859f399a7b84de99658380bd6c24ff5 (DEFAULT_TABLE table=default.Table_8d37b99f76eb49c8af370ba05e4015aa [id=27acc786d57a4c7ab531242dd52f17ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.298645 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 138) took 3103 ms. Trace:
I20250625 13:51:57.299870 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 with delete_type TABLET_DATA_DELETED (replaced by tablet f00e72e103f443e886f0df690cff4b6f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.301359 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e4bd4aaab143ffa65a668a952ae345. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.301664 1170 ts_tablet_manager.cc:1428] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 13:51:57.303086 1171 raft_consensus.cc:383] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:57.303200 1171 raft_consensus.cc:738] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:57.303483 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.303566 1171 consensus_queue.cc:260] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:57.304211 1171 ts_tablet_manager.cc:1428] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250625 13:51:57.304836 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 99090fee41d44946b6243b1b4bff3635 (table default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]) successfully deleted
W20250625 13:51:57.305913 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8fdca99b734448118e0c42227c78e58f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fdca99b734448118e0c42227c78e58f already in progress: creating tablet
I20250625 13:51:57.306159 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe4046742b34f1ab0b1610b5d6ee01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.307499 1465 raft_consensus.cc:491] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.307694 1465 raft_consensus.cc:513] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.307900 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 2c1a98dc40684954a602567cf3476879 (table default.Table_1231c5f6367b4046beaa57696ea42ff4 [id=97fc76973f074d29903fa25a80ca534a]) successfully deleted
I20250625 13:51:57.308530 1465 leader_election.cc:290] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.308781 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 (DEFAULT_TABLE table=default.Table_3e7baa9cc87143b0833a6bc32a193408 [id=d043c74a6cb94efa89ce819d2ef35545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.309585 896 tablet_service.cc:1468] Processing CreateTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 (DEFAULT_TABLE table=default.Table_3e7baa9cc87143b0833a6bc32a193408 [id=d043c74a6cb94efa89ce819d2ef35545]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:57.310497 985 leader_election.cc:343] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a21335a06ec04da29b3351554c75f03c
I20250625 13:51:57.310847 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a21335a06ec04da29b3351554c75f03c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:57.311098 913 raft_consensus.cc:2466] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:57.311605 983 leader_election.cc:304] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:57.311971 1023 tablet_service.cc:1468] Processing CreateTablet for tablet dfaba56442c0498da95e6fad126ab86d (DEFAULT_TABLE table=default.Table_eaf03b2c45a842c392e299f6fe84dc05 [id=7270b5aa9a0148178683a229db5b4b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.312181 1465 raft_consensus.cc:2802] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:57.312276 1465 raft_consensus.cc:491] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:57.312387 1465 raft_consensus.cc:3058] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.313182 894 tablet_service.cc:1468] Processing CreateTablet for tablet dfaba56442c0498da95e6fad126ab86d (DEFAULT_TABLE table=default.Table_eaf03b2c45a842c392e299f6fe84dc05 [id=7270b5aa9a0148178683a229db5b4b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.314705 1025 tablet_service.cc:1468] Processing CreateTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 (DEFAULT_TABLE table=default.Table_dc86aa488ba944c689ae858bc3800718 [id=028f5254a2c045929bdb41c0b273875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.315385 892 tablet_service.cc:1468] Processing CreateTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 (DEFAULT_TABLE table=default.Table_dc86aa488ba944c689ae858bc3800718 [id=028f5254a2c045929bdb41c0b273875c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.315846 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf7fab53cae49908d2952cdaa07943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.317378 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3859f399a7b84de99658380bd6c24ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.318733 916 raft_consensus.cc:2466] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:57.319993 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 9a2007e51d424053bc2836dedf46add2 (DEFAULT_TABLE table=default.Table_0568e1a3b89f4380abb0d3897817ccef [id=fa06a5b0571b4a3a8b9b68f1f05fc3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.320593 740 tablet_service.cc:1468] Processing CreateTablet for tablet f3c75deb64c74f0282efeed827659ec2 (DEFAULT_TABLE table=default.Table_32f5fbfdcc3840b8a4ff1122ff81208d [id=b4c1ac962c044cc79f9a8207fee04046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.320667 1465 raft_consensus.cc:513] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.321730 753 tablet_service.cc:1468] Processing CreateTablet for tablet 647c97a8ae484c4a8694e777bc62d602 (DEFAULT_TABLE table=default.Table_f9b2ff84e19a476bb879930b39eaa142 [id=49c81f046b034d91873a1051786c1c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.322196 757 tablet_service.cc:1468] Processing CreateTablet for tablet ef2b08fba37f4a38880c018ec2d869c8 (DEFAULT_TABLE table=default.Table_844f1dcd52aa496eb8c9c1520e7227d1 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:57.322271 985 leader_election.cc:343] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a21335a06ec04da29b3351554c75f03c
I20250625 13:51:57.323025 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 569f6cba69394f2fa005686056f19929 (DEFAULT_TABLE table=default.Table_06d8369e7a09451eba488006b53da953 [id=421fdfc88bfe40349eae76010c55e7ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.323398 1465 leader_election.cc:290] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.323470 745 tablet_service.cc:1468] Processing CreateTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 (DEFAULT_TABLE table=default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 [id=5b767eccaaaa46e0a054dd083e181cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.323734 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a21335a06ec04da29b3351554c75f03c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:57.323985 916 raft_consensus.cc:3058] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.324034 743 tablet_service.cc:1468] Processing CreateTablet for tablet d9462da7574a46839763b1c388102e60 (DEFAULT_TABLE table=default.Table_866deeb74a8743f0a579513bf580c9e9 [id=ce6cacfdab30448687a65de93031ab3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.324573 747 tablet_service.cc:1468] Processing CreateTablet for tablet fd164b54c1d14ea98422360705cf42bf (DEFAULT_TABLE table=default.Table_8b7b90e3ff0f4b81bef04d3952bae1ab [id=68f372185ad54993b8fd761fdd3d7592]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.324777 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 (DEFAULT_TABLE table=default.Table_89e73d4233d14f7ab1805e3351155943 [id=fe513f20709d46d3832d046a1544c47f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.326289 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd164b54c1d14ea98422360705cf42bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.327046 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2b08fba37f4a38880c018ec2d869c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.327800 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8985f259eb54ffdab29a9f6a2c5ae16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.328425 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9462da7574a46839763b1c388102e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.328922 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 647c97a8ae484c4a8694e777bc62d602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.329941 752 tablet_service.cc:1468] Processing CreateTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 (DEFAULT_TABLE table=default.Table_78c7bbeb91584e54bef556a0af56c2a7 [id=5602ea7a0c724767b8b53d6998acef30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.330303 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7506cce0ec4c4ec68f25591cb98b1252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.333976 895 tablet_service.cc:1468] Processing CreateTablet for tablet 9a2007e51d424053bc2836dedf46add2 (DEFAULT_TABLE table=default.Table_0568e1a3b89f4380abb0d3897817ccef [id=fa06a5b0571b4a3a8b9b68f1f05fc3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.335027 1171 tablet_bootstrap.cc:492] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:57.337987 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ef50a3624b4c79acc803ff0e2ed364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.325980 1168 rpcz_store.cc:276] 0625 13:51:54.181222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.181267 (+ 45us) service_pool.cc:225] Handling call
0625 13:51:57.280492 (+3099225us) ts_tablet_manager.cc:1145] Deleting tablet 799f2edd960d45c1ace9195283a042e2
0625 13:51:57.280504 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.284939 (+ 4435us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3099056}
I20250625 13:51:57.341480 747 tablet_service.cc:1468] Processing CreateTablet for tablet 32a57268a18f4f00a39f8360f9c90737 (DEFAULT_TABLE table=default.Table_61827a08a660423a91ddac5834f61382 [id=41c5c48c85234a85aa9ab76ba3f4c5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.341945 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a57268a18f4f00a39f8360f9c90737. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.342913 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c75deb64c74f0282efeed827659ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.349494 752 tablet_service.cc:1468] Processing CreateTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 (DEFAULT_TABLE table=default.Table_12771e585c3348049cc0ab4bf2a95935 [id=5eed1277a07641b6abc65ef047b31615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.349933 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413afd7b62d24fe8b9eceadd57e29fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.350754 891 tablet_service.cc:1468] Processing CreateTablet for tablet dec08d20a09444f5b9a102a8712e36c7 (DEFAULT_TABLE table=default.Table_8ba86fb74c0940fcaec4f753e8139f4a [id=d29821c37b644033be9a32f6adfb9710]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.355201 758 tablet_service.cc:1468] Processing CreateTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 (DEFAULT_TABLE table=default.Table_e7bc36f60d844f0d9d27755d6f58f63b [id=0a7bf2d0fdb148d2a201bddf8702be9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.355598 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca3f3760e6e41a29d004e6afe63ec98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.359136 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfaba56442c0498da95e6fad126ab86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.359853 1165 ts_tablet_manager.cc:1918] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:57.359990 1165 log.cc:1199] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/13620f6ca65342b4ad7f1ba4b437bde3
I20250625 13:51:57.360438 1165 ts_tablet_manager.cc:1939] T 13620f6ca65342b4ad7f1ba4b437bde3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:57.361323 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 268) took 1159 ms. Trace:
I20250625 13:51:57.361423 1165 rpcz_store.cc:276] 0625 13:51:56.201817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.201870 (+ 53us) service_pool.cc:225] Handling call
0625 13:51:57.188507 (+986637us) ts_tablet_manager.cc:1145] Deleting tablet 13620f6ca65342b4ad7f1ba4b437bde3
0625 13:51:57.192690 (+ 4183us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.258679 (+ 65989us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.359759 (+101080us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.361262 (+ 1503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":986361}
I20250625 13:51:57.361915 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 13620f6ca65342b4ad7f1ba4b437bde3 (table default.Table_6199817767f74031b8a500d21174d5a6 [id=bbf70badd2ea45ae88881afe31a3de40]) successfully deleted
I20250625 13:51:57.364131 887 tablet_service.cc:1468] Processing CreateTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 (DEFAULT_TABLE table=default.Table_89e73d4233d14f7ab1805e3351155943 [id=fe513f20709d46d3832d046a1544c47f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.365697 758 tablet_service.cc:1468] Processing CreateTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e (DEFAULT_TABLE table=default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.366174 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db6e9db2375144b38ec43ffca9ca4c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.366150 884 tablet_service.cc:1468] Processing CreateTablet for tablet b7d888e628a94f939ecf2571e60b1948 (DEFAULT_TABLE table=default.Table_7a2ca389ecd148b9be0e90c130fe6431 [id=26d0bdba8be543d89d5337c86bbf6e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.368481 1168 tablet_replica.cc:331] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:57.370077 1168 raft_consensus.cc:2241] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:57.370424 1168 pending_rounds.cc:70] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:51:57.371140 883 tablet_service.cc:1468] Processing CreateTablet for tablet 569f6cba69394f2fa005686056f19929 (DEFAULT_TABLE table=default.Table_06d8369e7a09451eba488006b53da953 [id=421fdfc88bfe40349eae76010c55e7ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.371978 1168 pending_rounds.cc:77] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520579408949248 op_type: NO_OP noop_request { }
I20250625 13:51:57.372179 1168 raft_consensus.cc:2887] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:57.373353 1168 raft_consensus.cc:2270] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:57.378734 581 consensus_peers.cc:487] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 -> Peer 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015): Couldn't send request to peer 3fec9d8a79054a5991ae90d8a3101e32. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250625 13:51:57.380584 1168 ts_tablet_manager.cc:1905] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:57.380684 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 799f2edd960d45c1ace9195283a042e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 799f2edd960d45c1ace9195283a042e2 already in progress: creating tablet
I20250625 13:51:57.384346 890 tablet_service.cc:1468] Processing CreateTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 (DEFAULT_TABLE table=default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.385378 885 tablet_service.cc:1468] Processing CreateTablet for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (DEFAULT_TABLE table=default.Table_25705d30e7034c57be006958705ab688 [id=3977ee3968d246b7a8aed1fafd1822b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.385738 1171 tablet_bootstrap.cc:654] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:57.386276 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2f8d0d9a714a28ba6fb9642d461de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.389765 916 raft_consensus.cc:2466] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:57.390630 983 leader_election.cc:304] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:57.391441 1465 raft_consensus.cc:2802] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:57.387686 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 269) took 1138 ms. Trace:
I20250625 13:51:57.391630 1165 rpcz_store.cc:276] 0625 13:51:56.249605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.249657 (+ 52us) service_pool.cc:225] Handling call
0625 13:51:57.387508 (+1137851us) ts_tablet_manager.cc:1145] Deleting tablet 1f6be7189f8f45ccb329ef5d2fd8da8d
0625 13:51:57.387518 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.387628 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134555}
W20250625 13:51:57.392154 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f6be7189f8f45ccb329ef5d2fd8da8d already in progress: creating tablet
I20250625 13:51:57.392949 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 270) took 1142 ms. Trace:
I20250625 13:51:57.393049 1165 rpcz_store.cc:276] 0625 13:51:56.250098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.250143 (+ 45us) service_pool.cc:225] Handling call
0625 13:51:57.392780 (+1142637us) ts_tablet_manager.cc:1145] Deleting tablet 36b151e200e1430c99ac194bc55881c1
0625 13:51:57.392789 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.392893 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1139706}
W20250625 13:51:57.393458 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 36b151e200e1430c99ac194bc55881c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36b151e200e1430c99ac194bc55881c1 already in progress: creating tablet
I20250625 13:51:57.395582 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 271) took 1144 ms. Trace:
I20250625 13:51:57.395696 1165 rpcz_store.cc:276] 0625 13:51:56.250632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.250676 (+ 44us) service_pool.cc:225] Handling call
0625 13:51:57.395405 (+1144729us) ts_tablet_manager.cc:1145] Deleting tablet bd5ba76c130a4711af5b21e9eb8562e7
0625 13:51:57.395416 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.395520 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142251}
W20250625 13:51:57.396118 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bd5ba76c130a4711af5b21e9eb8562e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5ba76c130a4711af5b21e9eb8562e7 already in progress: creating tablet
I20250625 13:51:57.397246 1465 raft_consensus.cc:695] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:57.398325 1165 tablet_replica.cc:331] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:57.398535 1165 raft_consensus.cc:2241] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:57.398828 1165 pending_rounds.cc:70] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:57.398895 1165 pending_rounds.cc:77] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520579830603776 op_type: NO_OP noop_request { }
I20250625 13:51:57.399031 1165 raft_consensus.cc:2887] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:57.399137 1165 raft_consensus.cc:2270] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:57.398288 1465 consensus_queue.cc:237] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.401829 1165 ts_tablet_manager.cc:1905] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:57.403378 711 leader_election.cc:343] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:57.403733 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 251288us
W20250625 13:51:57.404067 711 leader_election.cc:343] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:57.404384 711 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.407568 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dec08d20a09444f5b9a102a8712e36c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.407765 889 tablet_service.cc:1468] Processing CreateTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 (DEFAULT_TABLE table=default.Table_531f237d57114b2483ae3cf8d4b94659 [id=a525a4b043274f8ea88049ed7d1cf652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.412624 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_72d027cf329f419288cc0ad451b2b249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.413214 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72d027cf329f419288cc0ad451b2b249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.414446 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7e1e10c14c834d1ea94c3310c244e9d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.414587 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_248a6183f6d44e43ab1fa4722ec9a52e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.414965 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e1e10c14c834d1ea94c3310c244e9d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:57.414968 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_248a6183f6d44e43ab1fa4722ec9a52e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.415791 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6e6214b878ce4ee6b6152eda7b37e76c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.416218 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e6214b878ce4ee6b6152eda7b37e76c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.417171 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c3f75eb1454e43e2a012dad91ae07d37" } new_table_name: "default.Table_c1c9eece77144878a55acc01667331a9" modify_external_catalogs: true
I20250625 13:51:57.418346 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0231e5cbd04647a9911175976259ce4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.418787 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0231e5cbd04647a9911175976259ce4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.419905 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9e22da664adb404cad353db7745d0110" } new_table_name: "default.Table_1ffafeb50d654290963bf41dfeb57f55" modify_external_catalogs: true
I20250625 13:51:57.420135 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69672579a83c41d09ee7c78a6fa7d746. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.423226 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_e0ded9bf53314d66a9363d735e000f5c [id=a249a45df8b3454894e648d70f5e8b1f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dad4ea8cf44944b08f53428c95809daf consensus state has no leader
I20250625 13:51:57.423684 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 87327f74beb845768bbfd5ae6d2bcf88 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:57.429277 886 tablet_service.cc:1468] Processing CreateTablet for tablet a30a35e3639a43c5b078169060824723 (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.434736 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de9656958164dc4ae2fb0078b38f6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.435209 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c605d5b64c164c1594b0a40cfa2c4cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.435528 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30a35e3639a43c5b078169060824723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.437896 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_581fcddbe9bb428d84556e6156637b4a" } modify_external_catalogs: true
I20250625 13:51:57.438109 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_581fcddbe9bb428d84556e6156637b4a" } modify_external_catalogs: true
I20250625 13:51:57.439991 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9dfb8c48d5144c808567a16a30cc199e" } modify_external_catalogs: true
I20250625 13:51:57.440186 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9dfb8c48d5144c808567a16a30cc199e" } modify_external_catalogs: true
I20250625 13:51:57.441608 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba" } modify_external_catalogs: true
I20250625 13:51:57.441788 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba" } modify_external_catalogs: true
I20250625 13:51:57.447346 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 9ba63184ca2b4badafcf08309ef75164 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:57.453943 711 leader_election.cc:343] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: Tablet not RUNNING: STOPPED
I20250625 13:51:57.454126 711 leader_election.cc:304] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:51:57.454597 711 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.454722 1302 raft_consensus.cc:2747] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.454767 711 leader_election.cc:304] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:57.455047 1302 raft_consensus.cc:2747] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.455169 711 leader_election.cc:304] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:57.455569 711 leader_election.cc:304] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:57.455822 1373 raft_consensus.cc:2747] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.456032 1373 raft_consensus.cc:2802] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:57.456172 1373 raft_consensus.cc:695] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:57.456462 1373 consensus_queue.cc:237] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:57.460367 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 consensus state has no leader
I20250625 13:51:57.462497 1165 ts_tablet_manager.cc:1918] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:57.462742 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ea8cb48993044c799983dc051e437621" } new_table_name: "default.Table_ee7cfdccf06f4d94982b3b66c9bf1a1d" modify_external_catalogs: true
I20250625 13:51:57.463243 1165 log.cc:1199] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/cbaa018753d8455199f3869323fb5582
I20250625 13:51:57.463696 1165 ts_tablet_manager.cc:1939] T cbaa018753d8455199f3869323fb5582 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:57.464056 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6bb95cabd7b940cdafc8a4978beeac70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.464498 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 272) took 1192 ms. Trace:
W20250625 13:51:57.464531 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bb95cabd7b940cdafc8a4978beeac70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.464588 1165 rpcz_store.cc:276] 0625 13:51:56.272278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.274482 (+ 2204us) service_pool.cc:225] Handling call
0625 13:51:57.398240 (+1123758us) ts_tablet_manager.cc:1145] Deleting tablet cbaa018753d8455199f3869323fb5582
0625 13:51:57.398251 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.405220 (+ 6969us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.462363 (+ 57143us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.464432 (+ 2069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123560}
I20250625 13:51:57.465060 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet cbaa018753d8455199f3869323fb5582 (table default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]) successfully deleted
I20250625 13:51:57.466802 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ef50a3624b4c79acc803ff0e2ed364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.467069 1013 tablet_service.cc:1468] Processing CreateTablet for tablet dec08d20a09444f5b9a102a8712e36c7 (DEFAULT_TABLE table=default.Table_8ba86fb74c0940fcaec4f753e8139f4a [id=d29821c37b644033be9a32f6adfb9710]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.467470 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dec08d20a09444f5b9a102a8712e36c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.470316 1018 tablet_service.cc:1468] Processing CreateTablet for tablet b7d888e628a94f939ecf2571e60b1948 (DEFAULT_TABLE table=default.Table_7a2ca389ecd148b9be0e90c130fe6431 [id=26d0bdba8be543d89d5337c86bbf6e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.470695 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7d888e628a94f939ecf2571e60b1948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.470702 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_42f697776fd04dc3ab9be41c229289fe" } new_table_name: "default.Table_84313e8427b643219197cc4ad05fca40" modify_external_catalogs: true
I20250625 13:51:57.471961 1017 tablet_service.cc:1468] Processing CreateTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 (DEFAULT_TABLE table=default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.472311 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c605d5b64c164c1594b0a40cfa2c4cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.473155 1373 raft_consensus.cc:491] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.473331 1373 raft_consensus.cc:513] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:57.473934 1373 leader_election.cc:290] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:57.474363 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dbf3eb88d817489eae63201ef32a702c" } new_table_name: "default.Table_df6a42eba40144cf94d09e54694a85b9" modify_external_catalogs: true
I20250625 13:51:57.474429 1020 tablet_service.cc:1468] Processing CreateTablet for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (DEFAULT_TABLE table=default.Table_25705d30e7034c57be006958705ab688 [id=3977ee3968d246b7a8aed1fafd1822b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.474771 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c6a325d40c4f7ba2f342e39b9437e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.475430 1012 tablet_service.cc:1468] Processing CreateTablet for tablet a30a35e3639a43c5b078169060824723 (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.475746 1165 tablet_replica.cc:331] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:57.475766 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30a35e3639a43c5b078169060824723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.475945 1165 raft_consensus.cc:2241] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:57.476122 1165 raft_consensus.cc:2270] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:57.476675 1165 ts_tablet_manager.cc:1905] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:57.483855 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2007e51d424053bc2836dedf46add2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.486366 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d7e04ac5a7b3491bb209bc40967a3059" } modify_external_catalogs: true
I20250625 13:51:57.486554 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d7e04ac5a7b3491bb209bc40967a3059" } modify_external_catalogs: true
I20250625 13:51:57.489008 1165 ts_tablet_manager.cc:1918] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:57.492805 1165 log.cc:1199] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/5786bb7716ef44689ef9862fc3ce17e9
I20250625 13:51:57.493963 1165 ts_tablet_manager.cc:1939] T 5786bb7716ef44689ef9862fc3ce17e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:51:57.495007 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8063122ea7b24b6790f5c3cc884e09be consensus state has no leader
I20250625 13:51:57.496060 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 273) took 1219 ms. Trace:
I20250625 13:51:57.496330 1165 rpcz_store.cc:276] 0625 13:51:56.276881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.276936 (+ 55us) service_pool.cc:225] Handling call
0625 13:51:57.475652 (+1198716us) ts_tablet_manager.cc:1145] Deleting tablet 5786bb7716ef44689ef9862fc3ce17e9
0625 13:51:57.475666 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.481710 (+ 6044us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.488861 (+ 7151us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.496016 (+ 7155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198480}
I20250625 13:51:57.496667 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ac4d368347444846880954ced9651404
I20250625 13:51:57.497433 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a22ade8cefac4140b08882c311ea5181
I20250625 13:51:57.498205 740 tablet_service.cc:1515] Processing DeleteTablet for tablet ac4d368347444846880954ced9651404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.498214 743 tablet_service.cc:1515] Processing DeleteTablet for tablet a22ade8cefac4140b08882c311ea5181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.498764 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 5786bb7716ef44689ef9862fc3ce17e9 (table default.Table_180b708af9884741956d0cb3898f9efd [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]) successfully deleted
I20250625 13:51:57.499168 1165 tablet_replica.cc:331] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:57.499778 1165 raft_consensus.cc:2241] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:57.500730 1165 raft_consensus.cc:2270] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:57.501400 1165 ts_tablet_manager.cc:1905] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:57.501750 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c3f0ca6fa5482db7f57143911c838f consensus state has no leader
W20250625 13:51:57.502015 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 723246e1e6824faf9f9d39c3ea941fe4 consensus state has no leader
I20250625 13:51:57.503595 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 27092991fcad475fabc77d1f8bab0582
I20250625 13:51:57.503989 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9ba63184ca2b4badafcf08309ef75164
W20250625 13:51:57.504113 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 055f88e118454e32baa578cae9aceb0c consensus state has no leader
I20250625 13:51:57.504377 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 27092991fcad475fabc77d1f8bab0582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.504426 592 catalog_manager.cc:6652] ReplaceTablet: tablet 9ba63184ca2b4badafcf08309ef75164 deleted and replaced by tablet 829221b8a79f460d90dd24be6a2d4c43
I20250625 13:51:57.505059 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ba63184ca2b4badafcf08309ef75164 with delete_type TABLET_DATA_DELETED (replaced by tablet 829221b8a79f460d90dd24be6a2d4c43) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.505296 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ec42501f607c41ceb9d35f3dd2a38f34" } modify_external_catalogs: true
I20250625 13:51:57.505499 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ec42501f607c41ceb9d35f3dd2a38f34" } modify_external_catalogs: true
I20250625 13:51:57.505497 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_30da1f016b424e85bcbae236fafd8060"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.506150 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_64ba301a940e4256a5cc699fbbcbaef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.506506 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30da1f016b424e85bcbae236fafd8060 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.506670 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6da69facc8a3467297ae557c14575561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.507169 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da69facc8a3467297ae557c14575561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:57.508169 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64ba301a940e4256a5cc699fbbcbaef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.509444 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.509866 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.510257 1525 tablet.cc:1722] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:51:57.520103 888 tablet_service.cc:1515] Processing DeleteTablet for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.521409 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7d888e628a94f939ecf2571e60b1948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.524276 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a0e81d4c904248eba858462bcf31c5d9
I20250625 13:51:57.532886 1165 ts_tablet_manager.cc:1918] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:57.539153 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 02f2a2b595bc43309a29b3f1bc5d3766 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:57.539985 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 231) took 1826 ms. Trace:
I20250625 13:51:57.549947 1165 log.cc:1199] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/4b549bc8cead4af4a72f41f51655602c
I20250625 13:51:57.550336 743 tablet_service.cc:1515] Processing DeleteTablet for tablet a0e81d4c904248eba858462bcf31c5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.551123 740 tablet_service.cc:1468] Processing CreateTablet for tablet 669de0b7c93a4efea8623dab2e462c8d (DEFAULT_TABLE table=default.Table_72d027cf329f419288cc0ad451b2b249 [id=d3092ac7d43a4e9891fe289b304e937c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.551790 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669de0b7c93a4efea8623dab2e462c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.552021 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.552184 1166 tablet_bootstrap.cc:492] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:57.552368 1166 ts_tablet_manager.cc:1397] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.266s user 0.013s sys 0.010s
I20250625 13:51:57.555379 1166 raft_consensus.cc:357] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:57.555609 1165 ts_tablet_manager.cc:1939] T 4b549bc8cead4af4a72f41f51655602c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:57.555682 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d7704b38144429b9c95675be84a4730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.555989 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 423ca098138c476f861a19b44554fd24 (DEFAULT_TABLE table=default.Table_81732e5a747642cd8d61556c2ed6f6c7 [id=3f116c6f6ae94239941fa04a335853ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.556715 1166 raft_consensus.cc:383] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:57.558571 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.558828 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.558997 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.559562 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 (DEFAULT_TABLE table=default.Table_ba435656502a49659e22a61038cad9eb [id=56d2c709e389405c80e4fa20ce75d5ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.560395 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 (DEFAULT_TABLE table=default.Table_531f237d57114b2483ae3cf8d4b94659 [id=a525a4b043274f8ea88049ed7d1cf652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.560745 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2007e51d424053bc2836dedf46add2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.561708 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d26820eaba98401da3a8b5348be21fc6 (DEFAULT_TABLE table=default.Table_65410f67a07044e9a0b308de1bd53c34 [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.562373 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c6a325d40c4f7ba2f342e39b9437e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.562956 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 38770320fc6541c39e10448978f8a3b0 (DEFAULT_TABLE table=default.Table_3c72767e52f24da8b58c00aa584d15c6 [id=43487ee990e640bcbeefc7c142075998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.563586 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca (DEFAULT_TABLE table=default.Table_06ce856b6c374443991c4f9ce9a60eb1 [id=b36fcb27ef404ffeaa1532ad1063dbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.564749 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.565124 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 647c97a8ae484c4a8694e777bc62d602 (DEFAULT_TABLE table=default.Table_f9b2ff84e19a476bb879930b39eaa142 [id=49c81f046b034d91873a1051786c1c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.566169 1518 raft_consensus.cc:491] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.566885 1518 raft_consensus.cc:513] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.567643 1518 leader_election.cc:290] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.567796 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8149aea0ff7646d3b94c9159f56e48ef" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:57.567909 1167 rpcz_store.cc:276] 0625 13:51:55.713576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.715546 (+ 1970us) service_pool.cc:225] Handling call
0625 13:51:57.022647 (+1307101us) ts_tablet_manager.cc:1145] Deleting tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d
0625 13:51:57.022663 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.200949 (+178286us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.251220 (+ 50271us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.539935 (+288715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1306884}
I20250625 13:51:57.568346 1170 tablet_bootstrap.cc:492] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:57.568385 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 232) took 1854 ms. Trace:
I20250625 13:51:57.568509 1167 rpcz_store.cc:276] 0625 13:51:55.713683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.715790 (+ 2107us) service_pool.cc:225] Handling call
0625 13:51:57.568214 (+1852424us) ts_tablet_manager.cc:1145] Deleting tablet 8e29e7b55570483b80c6e7a1f227fa7b
0625 13:51:57.568223 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.568350 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852242}
I20250625 13:51:57.569200 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d (table default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e [id=2a70f0ade87f4f34bcbe23295f23234b]) successfully deleted
W20250625 13:51:57.569509 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8e29e7b55570483b80c6e7a1f227fa7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e29e7b55570483b80c6e7a1f227fa7b already in progress: creating tablet
I20250625 13:51:57.570300 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.571108 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.571818 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.571957 1014 tablet_service.cc:1468] Processing CreateTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 (DEFAULT_TABLE table=default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 [id=5b767eccaaaa46e0a054dd083e181cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.572129 1012 tablet_service.cc:1468] Processing CreateTablet for tablet d9462da7574a46839763b1c388102e60 (DEFAULT_TABLE table=default.Table_866deeb74a8743f0a579513bf580c9e9 [id=ce6cacfdab30448687a65de93031ab3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.572405 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 239) took 1801 ms. Trace:
I20250625 13:51:57.572783 1011 tablet_service.cc:1468] Processing CreateTablet for tablet ef2b08fba37f4a38880c018ec2d869c8 (DEFAULT_TABLE table=default.Table_844f1dcd52aa496eb8c9c1520e7227d1 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.572788 1022 tablet_service.cc:1468] Processing CreateTablet for tablet f3c75deb64c74f0282efeed827659ec2 (DEFAULT_TABLE table=default.Table_32f5fbfdcc3840b8a4ff1122ff81208d [id=b4c1ac962c044cc79f9a8207fee04046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.573329 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569f6cba69394f2fa005686056f19929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.573570 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c75deb64c74f0282efeed827659ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.574404 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de9656958164dc4ae2fb0078b38f6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.575217 1533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33015 every 8 connection(s)
I20250625 13:51:57.575362 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfaba56442c0498da95e6fad126ab86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.575541 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2b08fba37f4a38880c018ec2d869c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.575732 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2f8d0d9a714a28ba6fb9642d461de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.576840 1222 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33015
I20250625 13:51:57.578217 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1effed0fa10d409abcb31dbba54861d6
I20250625 13:51:57.579056 1222 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
I20250625 13:51:57.579401 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 02f2a2b595bc43309a29b3f1bc5d3766
I20250625 13:51:57.579852 590 catalog_manager.cc:6652] ReplaceTablet: tablet 02f2a2b595bc43309a29b3f1bc5d3766 deleted and replaced by tablet e3939ef22ea344328721dfba67d933d1
I20250625 13:51:57.583156 1167 rpcz_store.cc:276] 0625 13:51:55.768153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.768330 (+ 177us) service_pool.cc:225] Handling call
0625 13:51:57.568745 (+1800415us) ts_tablet_manager.cc:1145] Deleting tablet c0a3ea18bab944749d45da9b3139f8bc
0625 13:51:57.568753 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.569814 (+ 1061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1791327}
I20250625 13:51:57.583607 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 238) took 1854 ms. Trace:
I20250625 13:51:57.583705 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38770320fc6541c39e10448978f8a3b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.583783 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c0a3ea18bab944749d45da9b3139f8bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0a3ea18bab944749d45da9b3139f8bc already in progress: creating tablet
I20250625 13:51:57.588866 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423ca098138c476f861a19b44554fd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.589622 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2bc4df39674dc3be0cd32c68f1ebca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.590987 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077e8c001cc646559b5268d4a48c0ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.591173 896 tablet_service.cc:1468] Processing CreateTablet for tablet d26820eaba98401da3a8b5348be21fc6 (DEFAULT_TABLE table=default.Table_65410f67a07044e9a0b308de1bd53c34 [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.592011 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 1222
I20250625 13:51:57.592305 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8985f259eb54ffdab29a9f6a2c5ae16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.598445 1020 tablet_service.cc:1468] Processing CreateTablet for tablet fd164b54c1d14ea98422360705cf42bf (DEFAULT_TABLE table=default.Table_8b7b90e3ff0f4b81bef04d3952bae1ab [id=68f372185ad54993b8fd761fdd3d7592]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.598932 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd164b54c1d14ea98422360705cf42bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.602352 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69672579a83c41d09ee7c78a6fa7d746. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.602882 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26820eaba98401da3a8b5348be21fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.591923 1167 rpcz_store.cc:276] 0625 13:51:55.760650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.760781 (+ 131us) service_pool.cc:225] Handling call
0625 13:51:57.583452 (+1822671us) ts_tablet_manager.cc:1145] Deleting tablet 637847cac1724bfeb629c4d4013396ca
0625 13:51:57.583462 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.583565 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1804220}
W20250625 13:51:57.604050 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 637847cac1724bfeb629c4d4013396ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 637847cac1724bfeb629c4d4013396ca already in progress: creating tablet
I20250625 13:51:57.607728 1534 sys_catalog.cc:263] Verifying existing consensus state
I20250625 13:51:57.611210 1166 raft_consensus.cc:738] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:57.611555 1166 consensus_queue.cc:260] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:57.612162 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 1effed0fa10d409abcb31dbba54861d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.612418 1166 ts_tablet_manager.cc:1428] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20250625 13:51:57.613164 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c (DEFAULT_TABLE table=default.Table_30da1f016b424e85bcbae236fafd8060 [id=1f2d72b34f7a4cddb7555f2ddc708a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.613235 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 02f2a2b595bc43309a29b3f1bc5d3766 with delete_type TABLET_DATA_DELETED (replaced by tablet e3939ef22ea344328721dfba67d933d1) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.613539 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.613665 985 leader_election.cc:343] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 13620f6ca65342b4ad7f1ba4b437bde3
W20250625 13:51:57.613835 711 leader_election.cc:343] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:57.615342 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 44730.8us
I20250625 13:51:57.618585 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 274) took 1300 ms. Trace:
I20250625 13:51:57.618727 1165 rpcz_store.cc:276] 0625 13:51:56.312850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.312974 (+ 124us) service_pool.cc:225] Handling call
0625 13:51:57.499091 (+1186117us) ts_tablet_manager.cc:1145] Deleting tablet 4b549bc8cead4af4a72f41f51655602c
0625 13:51:57.499102 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.511951 (+ 12849us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.523157 (+ 11206us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.612789 (+ 89632us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1185917}
I20250625 13:51:57.618872 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 (DEFAULT_TABLE table=default.Table_248a6183f6d44e43ab1fa4722ec9a52e [id=85870f1c32df485aaef7d845e855b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.619782 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 4b549bc8cead4af4a72f41f51655602c (table default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]) successfully deleted
I20250625 13:51:57.620311 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9462da7574a46839763b1c388102e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.621634 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 647c97a8ae484c4a8694e777bc62d602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.622305 747 tablet_service.cc:1468] Processing CreateTablet for tablet 15c81fab8e0b442e8d39556c1bce742e (DEFAULT_TABLE table=default.Table_0231e5cbd04647a9911175976259ce4b [id=4d9dc7e6615940e38f7c61644a03022a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.624572 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 240) took 1843 ms. Trace:
I20250625 13:51:57.626405 1167 rpcz_store.cc:276] 0625 13:51:55.776921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.777114 (+ 193us) service_pool.cc:225] Handling call
0625 13:51:57.603677 (+1826563us) ts_tablet_manager.cc:1145] Deleting tablet 54fa042947da4e1b89018640b46b271a
0625 13:51:57.603687 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.620738 (+ 17051us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1822592}
W20250625 13:51:57.629989 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 54fa042947da4e1b89018640b46b271a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54fa042947da4e1b89018640b46b271a already in progress: creating tablet
I20250625 13:51:57.639690 1170 tablet_bootstrap.cc:654] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:57.637887 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 241) took 1852 ms. Trace:
I20250625 13:51:57.600477 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569f6cba69394f2fa005686056f19929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.607389 888 tablet_service.cc:1468] Processing CreateTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 (DEFAULT_TABLE table=default.Table_ba435656502a49659e22a61038cad9eb [id=56d2c709e389405c80e4fa20ce75d5ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.644100 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap starting.
I20250625 13:51:57.646673 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_44a75b53cab0470da32c4aedecdc2434"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.647502 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44a75b53cab0470da32c4aedecdc2434 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.646697 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b5f0cba34b2d436fbf0f6c029c7028dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.649587 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5f0cba34b2d436fbf0f6c029c7028dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.652632 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_974313f23d974d75914dc9b2f7bd25d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.653224 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_974313f23d974d75914dc9b2f7bd25d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.654951 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ceec0bc74ee04dd5b21dcb6735342a42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.655429 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceec0bc74ee04dd5b21dcb6735342a42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.657274 1167 rpcz_store.cc:276] 0625 13:51:55.777060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.777192 (+ 132us) service_pool.cc:225] Handling call
0625 13:51:57.627850 (+1850658us) ts_tablet_manager.cc:1145] Deleting tablet 58aa3c21191e4bea9c5395d6e32b8717
0625 13:51:57.627863 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.629204 (+ 1341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846224}
W20250625 13:51:57.658260 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 58aa3c21191e4bea9c5395d6e32b8717 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58aa3c21191e4bea9c5395d6e32b8717 already in progress: creating tablet
I20250625 13:51:57.661687 880 tablet_service.cc:1468] Processing CreateTablet for tablet 423ca098138c476f861a19b44554fd24 (DEFAULT_TABLE table=default.Table_81732e5a747642cd8d61556c2ed6f6c7 [id=3f116c6f6ae94239941fa04a335853ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.664626 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca (DEFAULT_TABLE table=default.Table_06ce856b6c374443991c4f9ce9a60eb1 [id=b36fcb27ef404ffeaa1532ad1063dbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.668362 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dacf434b035e41c293c9fb0057989436" } new_table_name: "default.Table_2a8634c60e08492faffd30a254d0dc11" modify_external_catalogs: true
I20250625 13:51:57.668538 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13620f6ca65342b4ad7f1ba4b437bde3" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:57.668799 916 raft_consensus.cc:2391] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:51:57.670682 1167 tablet_replica.cc:331] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:57.671085 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d7704b38144429b9c95675be84a4730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:51:57.672741 851 consensus_peers.cc:487] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5786bb7716ef44689ef9862fc3ce17e9. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:51:57.672856 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 59823us
W20250625 13:51:57.675873 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 490c8902efeb46c99543747a8787c1c7 consensus state has no leader
I20250625 13:51:57.677165 983 leader_election.cc:304] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:57.677695 1518 raft_consensus.cc:3058] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.677953 1534 log.cc:826] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Log is configured to *not* fsync() on all Append() calls
I20250625 13:51:57.681490 1167 raft_consensus.cc:2241] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:57.686839 893 tablet_service.cc:1468] Processing CreateTablet for tablet 38770320fc6541c39e10448978f8a3b0 (DEFAULT_TABLE table=default.Table_3c72767e52f24da8b58c00aa584d15c6 [id=43487ee990e640bcbeefc7c142075998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.688613 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7448453a4e8c44ab8ad28eef52b99d5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.689292 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7448453a4e8c44ab8ad28eef52b99d5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.690748 1167 raft_consensus.cc:2270] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:57.691820 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.692035 879 tablet_service.cc:1515] Processing DeleteTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.692715 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.692919 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.693913 879 tablet_service.cc:1468] Processing CreateTablet for tablet f3c75deb64c74f0282efeed827659ec2 (DEFAULT_TABLE table=default.Table_32f5fbfdcc3840b8a4ff1122ff81208d [id=b4c1ac962c044cc79f9a8207fee04046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.698611 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.699508 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fdde488d5ee94a8593bf9fc6205adeac" } modify_external_catalogs: true
I20250625 13:51:57.699779 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_068bedd508da462681c0138ddd8964e3" } modify_external_catalogs: true
I20250625 13:51:57.699962 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_068bedd508da462681c0138ddd8964e3" } modify_external_catalogs: true
I20250625 13:51:57.700457 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fdde488d5ee94a8593bf9fc6205adeac" } modify_external_catalogs: true
I20250625 13:51:57.703855 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_844f1dcd52aa496eb8c9c1520e7227d1" } new_table_name: "default.Table_8cf4529887f24853932e3c9a52a46b61" modify_external_catalogs: true
I20250625 13:51:57.704017 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3c72767e52f24da8b58c00aa584d15c6" } new_table_name: "default.Table_f49bf03b78c54f3eabcb74b4211ae036" modify_external_catalogs: true
I20250625 13:51:57.705987 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e6f0bb3d8be34d36a5c48eeb803e5053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.706550 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6f0bb3d8be34d36a5c48eeb803e5053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.709308 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_72d027cf329f419288cc0ad451b2b249" } new_table_name: "default.Table_550407055545488f84dc060ae1891f73" modify_external_catalogs: true
I20250625 13:51:57.710490 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b5f0cba34b2d436fbf0f6c029c7028dd" } modify_external_catalogs: true
I20250625 13:51:57.710667 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b5f0cba34b2d436fbf0f6c029c7028dd" } modify_external_catalogs: true
I20250625 13:51:57.712888 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cd4f0857f52449e1b26fde09c6c7c9d4
I20250625 13:51:57.713605 1167 ts_tablet_manager.cc:1905] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:57.715782 881 tablet_service.cc:1468] Processing CreateTablet for tablet 647c97a8ae484c4a8694e777bc62d602 (DEFAULT_TABLE table=default.Table_f9b2ff84e19a476bb879930b39eaa142 [id=49c81f046b034d91873a1051786c1c32]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:57.717703 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef2b08fba37f4a38880c018ec2d869c8 consensus state has no leader
W20250625 13:51:57.718601 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38770320fc6541c39e10448978f8a3b0 consensus state has no leader
I20250625 13:51:57.727053 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26820eaba98401da3a8b5348be21fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.727295 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e6f5787513b7437ab9fa137510881691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.728076 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6f5787513b7437ab9fa137510881691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.729341 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9234ef66a80b476b9614ca31733dfa74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.729880 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9234ef66a80b476b9614ca31733dfa74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.730388 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_61827a08a660423a91ddac5834f61382" } modify_external_catalogs: true
I20250625 13:51:57.730604 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_61827a08a660423a91ddac5834f61382" } modify_external_catalogs: true
I20250625 13:51:57.729372 877 tablet_service.cc:1468] Processing CreateTablet for tablet ef2b08fba37f4a38880c018ec2d869c8 (DEFAULT_TABLE table=default.Table_844f1dcd52aa496eb8c9c1520e7227d1 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:57.733472 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 669de0b7c93a4efea8623dab2e462c8d consensus state has no leader
I20250625 13:51:57.733880 593 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7260ae5fb487421994867f8313f0d597
I20250625 13:51:57.735015 889 tablet_service.cc:1468] Processing CreateTablet for tablet fd164b54c1d14ea98422360705cf42bf (DEFAULT_TABLE table=default.Table_8b7b90e3ff0f4b81bef04d3952bae1ab [id=68f372185ad54993b8fd761fdd3d7592]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.737986 1518 raft_consensus.cc:2747] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.738991 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077e8c001cc646559b5268d4a48c0ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.739602 882 tablet_service.cc:1468] Processing CreateTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 (DEFAULT_TABLE table=default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 [id=5b767eccaaaa46e0a054dd083e181cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.742020 892 tablet_service.cc:1468] Processing CreateTablet for tablet d9462da7574a46839763b1c388102e60 (DEFAULT_TABLE table=default.Table_866deeb74a8743f0a579513bf580c9e9 [id=ce6cacfdab30448687a65de93031ab3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.743140 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_45ffe0341a354703ab5ab6c66bd27f18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.735209 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9b1b976f4c234b97923637678b6508f9" } new_table_name: "default.Table_4dc8c488c3e44f3f882049d1c0f8feb2" modify_external_catalogs: true
I20250625 13:51:57.744369 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423ca098138c476f861a19b44554fd24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.746053 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2bc4df39674dc3be0cd32c68f1ebca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.746526 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45ffe0341a354703ab5ab6c66bd27f18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.747620 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a29eb5d240db4787866630bc39e35db5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.748337 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a29eb5d240db4787866630bc39e35db5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.749152 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8985f259eb54ffdab29a9f6a2c5ae16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.758831 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38770320fc6541c39e10448978f8a3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.761710 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 (DEFAULT_TABLE table=default.Table_78c7bbeb91584e54bef556a0af56c2a7 [id=5602ea7a0c724767b8b53d6998acef30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.762267 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7506cce0ec4c4ec68f25591cb98b1252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.763362 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d524089e6ab044ec961cfc9f587e8f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.763900 886 tablet_service.cc:1468] Processing CreateTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 (DEFAULT_TABLE table=default.Table_78c7bbeb91584e54bef556a0af56c2a7 [id=5602ea7a0c724767b8b53d6998acef30]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:57.764994 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2404e712e18146b68757bd77362c1358 consensus state has no leader
I20250625 13:51:57.766471 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 32a57268a18f4f00a39f8360f9c90737 (DEFAULT_TABLE table=default.Table_61827a08a660423a91ddac5834f61382 [id=41c5c48c85234a85aa9ab76ba3f4c5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.767398 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 (DEFAULT_TABLE table=default.Table_12771e585c3348049cc0ab4bf2a95935 [id=5eed1277a07641b6abc65ef047b31615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.768226 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c75deb64c74f0282efeed827659ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.778664 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8149aea0ff7646d3b94c9159f56e48ef" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:57.779054 890 tablet_service.cc:1468] Processing CreateTablet for tablet 32a57268a18f4f00a39f8360f9c90737 (DEFAULT_TABLE table=default.Table_61827a08a660423a91ddac5834f61382 [id=41c5c48c85234a85aa9ab76ba3f4c5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.784953 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3ac6f01366104d75a5bf908bc40c3f4f" } modify_external_catalogs: true
I20250625 13:51:57.785245 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3ac6f01366104d75a5bf908bc40c3f4f" } modify_external_catalogs: true
I20250625 13:51:57.787628 894 tablet_service.cc:1468] Processing CreateTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 (DEFAULT_TABLE table=default.Table_12771e585c3348049cc0ab4bf2a95935 [id=5eed1277a07641b6abc65ef047b31615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.788512 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 647c97a8ae484c4a8694e777bc62d602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.791009 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd164b54c1d14ea98422360705cf42bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.785257 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 (DEFAULT_TABLE table=default.Table_e7bc36f60d844f0d9d27755d6f58f63b [id=0a7bf2d0fdb148d2a201bddf8702be9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.791366 1022 tablet_service.cc:1468] Processing CreateTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e (DEFAULT_TABLE table=default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.795481 891 tablet_service.cc:1468] Processing CreateTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 (DEFAULT_TABLE table=default.Table_e7bc36f60d844f0d9d27755d6f58f63b [id=0a7bf2d0fdb148d2a201bddf8702be9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.793411 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet ac4d368347444846880954ced9651404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.796187 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca3f3760e6e41a29d004e6afe63ec98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.797032 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413afd7b62d24fe8b9eceadd57e29fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.801677 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a57268a18f4f00a39f8360f9c90737. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.802573 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9462da7574a46839763b1c388102e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.802973 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7506cce0ec4c4ec68f25591cb98b1252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.803236 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet a22ade8cefac4140b08882c311ea5181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.808708 883 tablet_service.cc:1468] Processing CreateTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e (DEFAULT_TABLE table=default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.809172 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db6e9db2375144b38ec43ffca9ca4c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.808348 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ba63184ca2b4badafcf08309ef75164 with delete_type TABLET_DATA_DELETED (replaced by tablet 829221b8a79f460d90dd24be6a2d4c43) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.807065 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 27092991fcad475fabc77d1f8bab0582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.812389 895 tablet_service.cc:1515] Processing DeleteTablet for tablet ac4d368347444846880954ced9651404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.812690 895 tablet_service.cc:1515] Processing DeleteTablet for tablet a22ade8cefac4140b08882c311ea5181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.812903 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 27092991fcad475fabc77d1f8bab0582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.813143 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ba63184ca2b4badafcf08309ef75164 with delete_type TABLET_DATA_DELETED (replaced by tablet 829221b8a79f460d90dd24be6a2d4c43) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.813555 895 tablet_service.cc:1468] Processing CreateTablet for tablet 15c81fab8e0b442e8d39556c1bce742e (DEFAULT_TABLE table=default.Table_0231e5cbd04647a9911175976259ce4b [id=4d9dc7e6615940e38f7c61644a03022a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.813951 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 32a57268a18f4f00a39f8360f9c90737
I20250625 13:51:57.814098 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c81fab8e0b442e8d39556c1bce742e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.815487 1166 tablet_bootstrap.cc:492] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:57.816711 758 tablet_service.cc:1468] Processing CreateTablet for tablet eafc402d809245d189f3977162506ef8 (DEFAULT_TABLE table=default.Table_6bb95cabd7b940cdafc8a4978beeac70 [id=e77b05be11de44639e5a6ae4a6367880]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.816975 1373 raft_consensus.cc:491] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:57.817132 1373 raft_consensus.cc:513] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.811014 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet a0e81d4c904248eba858462bcf31c5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.817778 1373 leader_election.cc:290] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.817961 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (DEFAULT_TABLE table=default.Table_7e1e10c14c834d1ea94c3310c244e9d6 [id=6903813257eb4459a5b89035cae9e85a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.818181 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3249c5c2aa564027a2c594ca4c31e1c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.819835 711 leader_election.cc:343] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.819957 1165 tablet_replica.cc:331] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:57.821031 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 37e4bd4aaab143ffa65a668a952ae345
I20250625 13:51:57.823148 711 leader_election.cc:304] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:57.824101 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:57.825026 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:57.825189 753 tablet_service.cc:1468] Processing CreateTablet for tablet df72a1a461aa44fe80cff6e3870da8e7 (DEFAULT_TABLE table=default.Table_64ba301a940e4256a5cc699fbbcbaef5 [id=1ca6396d145c4b68a68e153cf78f8bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.825253 916 raft_consensus.cc:2466] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
W20250625 13:51:57.825484 711 leader_election.cc:343] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.825515 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a57268a18f4f00a39f8360f9c90737 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.826428 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 68b8c8aa8036454298bf1e81bab91c01
I20250625 13:51:57.828713 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.829049 751 tablet_service.cc:1515] Processing DeleteTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.829188 755 tablet_service.cc:1468] Processing CreateTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 (DEFAULT_TABLE table=default.Table_dbca3ace8835418e8db2cbc1845d9cfb [id=1cd7ba4de230481e8b0c57c1c585fbd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.830057 741 tablet_service.cc:1468] Processing CreateTablet for tablet 4d2d6f05348e43339e7167a9cddca2ba (DEFAULT_TABLE table=default.Table_6e6214b878ce4ee6b6152eda7b37e76c [id=83f8d808e36c40deb230bff2a04e7c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.830262 1165 raft_consensus.cc:2241] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:57.818698 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 1effed0fa10d409abcb31dbba54861d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.831483 711 leader_election.cc:304] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:57.831769 1302 raft_consensus.cc:2802] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:57.832692 1302 raft_consensus.cc:491] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:57.832839 1302 raft_consensus.cc:3058] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.831233 1373 raft_consensus.cc:2747] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:57.839422 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c81fab8e0b442e8d39556c1bce742e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.836284 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 02f2a2b595bc43309a29b3f1bc5d3766 with delete_type TABLET_DATA_DELETED (replaced by tablet e3939ef22ea344328721dfba67d933d1) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.837369 749 tablet_service.cc:1468] Processing CreateTablet for tablet e3939ef22ea344328721dfba67d933d1 (DEFAULT_TABLE table=default.Table_aacf50f903c54d17a68e13722c931d22 [id=8c054766f5424144a4edb39b192451cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.837998 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 (DEFAULT_TABLE table=default.Table_248a6183f6d44e43ab1fa4722ec9a52e [id=85870f1c32df485aaef7d845e855b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.845477 880 tablet_service.cc:1515] Processing DeleteTablet for tablet a0e81d4c904248eba858462bcf31c5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.845860 880 tablet_service.cc:1468] Processing CreateTablet for tablet 669de0b7c93a4efea8623dab2e462c8d (DEFAULT_TABLE table=default.Table_72d027cf329f419288cc0ad451b2b249 [id=d3092ac7d43a4e9891fe289b304e937c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.846216 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669de0b7c93a4efea8623dab2e462c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.847414 752 tablet_service.cc:1468] Processing CreateTablet for tablet ed3c177bb6dd488aa54bee2a447016ff (DEFAULT_TABLE table=default.Table_6da69facc8a3467297ae557c14575561 [id=4631cf687a9c478ba42e9ea3375ca639]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.849505 878 tablet_service.cc:1468] Processing CreateTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 (DEFAULT_TABLE table=default.Table_248a6183f6d44e43ab1fa4722ec9a52e [id=85870f1c32df485aaef7d845e855b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.849830 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3249c5c2aa564027a2c594ca4c31e1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.851388 1165 raft_consensus.cc:2270] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:57.851800 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c (DEFAULT_TABLE table=default.Table_30da1f016b424e85bcbae236fafd8060 [id=1f2d72b34f7a4cddb7555f2ddc708a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.845243 1015 tablet_service.cc:1468] Processing CreateTablet for tablet eafc402d809245d189f3977162506ef8 (DEFAULT_TABLE table=default.Table_6bb95cabd7b940cdafc8a4978beeac70 [id=e77b05be11de44639e5a6ae4a6367880]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.856282 750 tablet_service.cc:1468] Processing CreateTablet for tablet afa1c198dd6541809843244c7cc18194 (DEFAULT_TABLE table=default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94 [id=3c2aabfc6e104537ac703fbcf59f2470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.857096 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:57.853673 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 15c81fab8e0b442e8d39556c1bce742e (DEFAULT_TABLE table=default.Table_0231e5cbd04647a9911175976259ce4b [id=4d9dc7e6615940e38f7c61644a03022a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.861771 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 1effed0fa10d409abcb31dbba54861d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.862008 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 02f2a2b595bc43309a29b3f1bc5d3766 with delete_type TABLET_DATA_DELETED (replaced by tablet e3939ef22ea344328721dfba67d933d1) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:57.862944 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0d3b87078b494559b4a686a10a478402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.863586 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d3b87078b494559b4a686a10a478402 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.864209 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:57.867401 1442 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.012s sys 0.014s
W20250625 13:51:57.867497 1442 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.870s user 0.012s sys 0.014s
I20250625 13:51:57.867938 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2b08fba37f4a38880c018ec2d869c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.868410 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (DEFAULT_TABLE table=default.Table_7e1e10c14c834d1ea94c3310c244e9d6 [id=6903813257eb4459a5b89035cae9e85a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.869300 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e78b8ab5d2584ab19f2560a2298b21cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.869823 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e78b8ab5d2584ab19f2560a2298b21cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.871508 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9b8e7cde4c874918a309446de3d87d66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.872138 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b8e7cde4c874918a309446de3d87d66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.874646 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e6080511c8eb46c086fdb07159149641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.875236 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6080511c8eb46c086fdb07159149641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.879962 1165 ts_tablet_manager.cc:1905] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:57.881309 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_25705d30e7034c57be006958705ab688" } new_table_name: "default.Table_cf5bc408ef7141fa9893bdf60c24d10b" modify_external_catalogs: true
I20250625 13:51:57.881394 756 tablet_service.cc:1515] Processing DeleteTablet for tablet d524089e6ab044ec961cfc9f587e8f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:57.881683 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2d6f05348e43339e7167a9cddca2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.889998 1013 tablet_service.cc:1468] Processing CreateTablet for tablet df72a1a461aa44fe80cff6e3870da8e7 (DEFAULT_TABLE table=default.Table_64ba301a940e4256a5cc699fbbcbaef5 [id=1ca6396d145c4b68a68e153cf78f8bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.890919 1017 tablet_service.cc:1468] Processing CreateTablet for tablet ed3c177bb6dd488aa54bee2a447016ff (DEFAULT_TABLE table=default.Table_6da69facc8a3467297ae557c14575561 [id=4631cf687a9c478ba42e9ea3375ca639]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.896590 1026 tablet_service.cc:1468] Processing CreateTablet for tablet afa1c198dd6541809843244c7cc18194 (DEFAULT_TABLE table=default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94 [id=3c2aabfc6e104537ac703fbcf59f2470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.900944 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a4182ebe832a4ef2a6b0a736bfcbc4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.901661 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4182ebe832a4ef2a6b0a736bfcbc4b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.901789 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 4d2d6f05348e43339e7167a9cddca2ba (DEFAULT_TABLE table=default.Table_6e6214b878ce4ee6b6152eda7b37e76c [id=83f8d808e36c40deb230bff2a04e7c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.909732 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa52e79e00c4f9c97a05c353e11ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.910014 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 (DEFAULT_TABLE table=default.Table_dbca3ace8835418e8db2cbc1845d9cfb [id=1cd7ba4de230481e8b0c57c1c585fbd2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:57.911612 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7c6a325d40c4f7ba2f342e39b9437e3 consensus state has no leader
I20250625 13:51:57.913242 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e3939ef22ea344328721dfba67d933d1 (DEFAULT_TABLE table=default.Table_aacf50f903c54d17a68e13722c931d22 [id=8c054766f5424144a4edb39b192451cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.914758 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20f874f96e2d4a3089c21ff64313c62a" } modify_external_catalogs: true
I20250625 13:51:57.914979 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20f874f96e2d4a3089c21ff64313c62a" } modify_external_catalogs: true
I20250625 13:51:57.916837 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed3c177bb6dd488aa54bee2a447016ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.917737 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.917941 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a57268a18f4f00a39f8360f9c90737 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.918068 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.918198 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:57.918619 1168 ts_tablet_manager.cc:1918] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:57.918589 1011 tablet_service.cc:1468] Processing CreateTablet for tablet e2e6fbf3d81547f798b246a8b8f61387 (DEFAULT_TABLE table=default.Table_974313f23d974d75914dc9b2f7bd25d9 [id=7172d4663e37479e9998f617d88f8b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.918740 1168 log.cc:1199] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/4b549bc8cead4af4a72f41f51655602c
I20250625 13:51:57.919181 1168 ts_tablet_manager.cc:1939] T 4b549bc8cead4af4a72f41f51655602c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:57.919341 756 tablet_service.cc:1468] Processing CreateTablet for tablet 71cf3b95139245a4ba11b6bcaa3ab3f4 (DEFAULT_TABLE table=default.Table_ceec0bc74ee04dd5b21dcb6735342a42 [id=b3427651d7264e36922da55c8d267546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.919557 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f2cf2aab62814c7389fa4b446b696dd1" } new_table_name: "default.Table_08c36a3d2a03417982751548b18def03" modify_external_catalogs: true
I20250625 13:51:57.921906 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6da69facc8a3467297ae557c14575561" } new_table_name: "default.Table_c4f1a08649504a7db750c88b65ff985c" modify_external_catalogs: true
I20250625 13:51:57.922039 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 71cf3b95139245a4ba11b6bcaa3ab3f4 (DEFAULT_TABLE table=default.Table_ceec0bc74ee04dd5b21dcb6735342a42 [id=b3427651d7264e36922da55c8d267546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.923431 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed3c177bb6dd488aa54bee2a447016ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.924741 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df72a1a461aa44fe80cff6e3870da8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.924728 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa1c198dd6541809843244c7cc18194. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.926035 751 tablet_service.cc:1468] Processing CreateTablet for tablet ef2c3757c0284aaca70ec317c05195d6 (DEFAULT_TABLE table=default.Table_44a75b53cab0470da32c4aedecdc2434 [id=7e8ccd792ea1452ba8c9fb719d11de52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.926499 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3939ef22ea344328721dfba67d933d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.929566 1166 tablet_bootstrap.cc:654] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:57.930958 740 tablet_service.cc:1468] Processing CreateTablet for tablet e2e6fbf3d81547f798b246a8b8f61387 (DEFAULT_TABLE table=default.Table_974313f23d974d75914dc9b2f7bd25d9 [id=7172d4663e37479e9998f617d88f8b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.931286 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2c3757c0284aaca70ec317c05195d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.932248 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e6fbf3d81547f798b246a8b8f61387. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.933075 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df72a1a461aa44fe80cff6e3870da8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.933262 1028 tablet_service.cc:1468] Processing CreateTablet for tablet ef2c3757c0284aaca70ec317c05195d6 (DEFAULT_TABLE table=default.Table_44a75b53cab0470da32c4aedecdc2434 [id=7e8ccd792ea1452ba8c9fb719d11de52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.933631 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2c3757c0284aaca70ec317c05195d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.933838 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829221b8a79f460d90dd24be6a2d4c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.934880 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafc402d809245d189f3977162506ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.935770 1302 raft_consensus.cc:513] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.936558 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_32f5fbfdcc3840b8a4ff1122ff81208d" } new_table_name: "default.Table_513b4c6ebe2942e1a4018f3d8e385853" modify_external_catalogs: true
I20250625 13:51:57.936836 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e6f5787513b7437ab9fa137510881691" } modify_external_catalogs: true
I20250625 13:51:57.937013 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e6f5787513b7437ab9fa137510881691" } modify_external_catalogs: true
I20250625 13:51:57.937271 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_58618cdd2a7e4f508dbbb9a221a34800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.937693 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71cf3b95139245a4ba11b6bcaa3ab3f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:57.937840 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58618cdd2a7e4f508dbbb9a221a34800 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.938478 1302 leader_election.cc:290] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:57.939049 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:51:57.939616 711 leader_election.cc:343] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:57.940057 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2d6f05348e43339e7167a9cddca2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.943507 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829221b8a79f460d90dd24be6a2d4c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.943909 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3947dfd6b9454b7eb988acee97d6e311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:57.944253 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_675853c72fc2452eaad0d0a7dd62fd55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.944487 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3947dfd6b9454b7eb988acee97d6e311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:57.944741 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_675853c72fc2452eaad0d0a7dd62fd55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.949232 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa1c198dd6541809843244c7cc18194. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.955561 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71cf3b95139245a4ba11b6bcaa3ab3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.958767 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d568530b5b6e472fb3c3fd4b301a6dd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:57.959497 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d568530b5b6e472fb3c3fd4b301a6dd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:57.963099 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3939ef22ea344328721dfba67d933d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.973747 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e6fbf3d81547f798b246a8b8f61387. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:57.981024 1545 tablet.cc:1722] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:51:57.983404 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:57.983634 916 raft_consensus.cc:3058] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:57.986359 887 tablet_service.cc:1468] Processing CreateTablet for tablet eafc402d809245d189f3977162506ef8 (DEFAULT_TABLE table=default.Table_6bb95cabd7b940cdafc8a4978beeac70 [id=e77b05be11de44639e5a6ae4a6367880]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.988274 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14a12a5453e647b9946ca19e0ffb14ee" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:57.991597 882 tablet_service.cc:1468] Processing CreateTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c (DEFAULT_TABLE table=default.Table_30da1f016b424e85bcbae236fafd8060 [id=1f2d72b34f7a4cddb7555f2ddc708a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.992889 1171 tablet_bootstrap.cc:492] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:57.993028 1171 ts_tablet_manager.cc:1397] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.689s user 0.005s sys 0.010s
I20250625 13:51:57.993760 1171 raft_consensus.cc:357] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.994326 1171 raft_consensus.cc:383] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:57.994444 1171 raft_consensus.cc:738] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:57.994689 1171 consensus_queue.cc:260] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:57.995343 1171 ts_tablet_manager.cc:1428] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:57.998364 1171 tablet_bootstrap.cc:492] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.001950 1170 tablet_bootstrap.cc:492] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:58.002192 1170 ts_tablet_manager.cc:1397] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.521s user 0.005s sys 0.009s
W20250625 13:51:58.003374 849 consensus_peers.cc:487] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf5a6951aba84f5fa8f6d76a2b7fcf2d. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:58.003436 1170 raft_consensus.cc:357] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:58.003723 849 leader_election.cc:343] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.003866 1170 raft_consensus.cc:383] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.003844 849 leader_election.cc:304] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:58.004108 1510 consensus_queue.cc:1035] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:58.004509 1510 raft_consensus.cc:2747] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250625 13:51:58.004747 1170 raft_consensus.cc:738] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.005088 1170 consensus_queue.cc:260] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.011083 881 tablet_service.cc:1468] Processing CreateTablet for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (DEFAULT_TABLE table=default.Table_7e1e10c14c834d1ea94c3310c244e9d6 [id=6903813257eb4459a5b89035cae9e85a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.020859 888 tablet_service.cc:1468] Processing CreateTablet for tablet df72a1a461aa44fe80cff6e3870da8e7 (DEFAULT_TABLE table=default.Table_64ba301a940e4256a5cc699fbbcbaef5 [id=1ca6396d145c4b68a68e153cf78f8bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.021941 893 tablet_service.cc:1468] Processing CreateTablet for tablet ed3c177bb6dd488aa54bee2a447016ff (DEFAULT_TABLE table=default.Table_6da69facc8a3467297ae557c14575561 [id=4631cf687a9c478ba42e9ea3375ca639]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.022264 1171 tablet_bootstrap.cc:654] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.023061 884 tablet_service.cc:1468] Processing CreateTablet for tablet afa1c198dd6541809843244c7cc18194 (DEFAULT_TABLE table=default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94 [id=3c2aabfc6e104537ac703fbcf59f2470]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.024453 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4d2d6f05348e43339e7167a9cddca2ba (DEFAULT_TABLE table=default.Table_6e6214b878ce4ee6b6152eda7b37e76c [id=83f8d808e36c40deb230bff2a04e7c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.025900 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4282c7f41da742dca77dc5965c226520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.026489 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4282c7f41da742dca77dc5965c226520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.027163 890 tablet_service.cc:1468] Processing CreateTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 (DEFAULT_TABLE table=default.Table_dbca3ace8835418e8db2cbc1845d9cfb [id=1cd7ba4de230481e8b0c57c1c585fbd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:57.839227 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 669de0b7c93a4efea8623dab2e462c8d (DEFAULT_TABLE table=default.Table_72d027cf329f419288cc0ad451b2b249 [id=d3092ac7d43a4e9891fe289b304e937c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.028357 894 tablet_service.cc:1468] Processing CreateTablet for tablet e3939ef22ea344328721dfba67d933d1 (DEFAULT_TABLE table=default.Table_aacf50f903c54d17a68e13722c931d22 [id=8c054766f5424144a4edb39b192451cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.029224 916 raft_consensus.cc:2466] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:58.037175 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ca3f3760e6e41a29d004e6afe63ec98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.038676 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669de0b7c93a4efea8623dab2e462c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.039376 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet a21335a06ec04da29b3351554c75f03c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.040454 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa52e79e00c4f9c97a05c353e11ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.040730 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3939ef22ea344328721dfba67d933d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.040957 896 tablet_service.cc:1515] Processing DeleteTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.041290 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829221b8a79f460d90dd24be6a2d4c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.042961 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df72a1a461aa44fe80cff6e3870da8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.044917 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed3c177bb6dd488aa54bee2a447016ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.045452 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafc402d809245d189f3977162506ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.046151 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.046648 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2d6f05348e43339e7167a9cddca2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.048990 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet db6e9db2375144b38ec43ffca9ca4c1e
I20250625 13:51:58.050310 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.050885 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.055991 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 139) took 3874 ms. Trace:
I20250625 13:51:58.056108 1168 rpcz_store.cc:276] 0625 13:51:54.181587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.181626 (+ 39us) service_pool.cc:225] Handling call
0625 13:51:57.340635 (+3159009us) ts_tablet_manager.cc:1145] Deleting tablet 4b549bc8cead4af4a72f41f51655602c
0625 13:51:57.355792 (+ 15157us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.913774 (+557982us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:57.918485 (+ 4711us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.055957 (+137472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3158842}
I20250625 13:51:58.057041 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 140) took 3874 ms. Trace:
I20250625 13:51:58.057145 1168 rpcz_store.cc:276] 0625 13:51:54.181961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.181999 (+ 38us) service_pool.cc:225] Handling call
0625 13:51:58.056395 (+3874396us) ts_tablet_manager.cc:1145] Deleting tablet b7641869a0bb463698244ff445d87a1f
0625 13:51:58.056403 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.056506 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874251}
I20250625 13:51:58.057416 1168 tablet_replica.cc:331] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:58.057636 1168 raft_consensus.cc:2241] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:58.057907 1168 pending_rounds.cc:70] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:51:58.057981 1168 pending_rounds.cc:77] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520579378835456 op_type: NO_OP noop_request { }
I20250625 13:51:58.058079 1168 raft_consensus.cc:2887] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:58.058178 1168 raft_consensus.cc:2270] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.058668 1168 ts_tablet_manager.cc:1905] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.065860 1442 raft_consensus.cc:491] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.066023 1442 raft_consensus.cc:513] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.066749 1442 leader_election.cc:290] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:58.068178 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5513b6e4edc14bdea44f9e8562b4d70e" } new_table_name: "default.Table_fe872dae10544c0ca6a6735266cd03d2" modify_external_catalogs: true
I20250625 13:51:58.071522 1167 ts_tablet_manager.cc:1918] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:51:58.081895 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4f19ddc51324cc18242c6081f3f7de5 consensus state has no leader
I20250625 13:51:58.086993 711 leader_election.cc:304] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
W20250625 13:51:58.090116 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 50522.4us
I20250625 13:51:58.090205 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d524089e6ab044ec961cfc9f587e8f86" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:58.092581 777 raft_consensus.cc:2466] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:58.093199 851 leader_election.cc:304] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:58.093392 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=5 ignored=0} inserts{seen=5 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 9 replicates
I20250625 13:51:58.093549 1442 raft_consensus.cc:2802] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:58.093709 1442 raft_consensus.cc:491] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:58.093803 1442 raft_consensus.cc:3058] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.097057 750 tablet_service.cc:1515] Processing DeleteTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.097033 1442 raft_consensus.cc:513] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.098137 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 4b549bc8cead4af4a72f41f51655602c (table default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]) successfully deleted
I20250625 13:51:58.098238 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d524089e6ab044ec961cfc9f587e8f86" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:51:58.098400 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet b7641869a0bb463698244ff445d87a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7641869a0bb463698244ff445d87a1f already in progress: creating tablet
W20250625 13:51:58.098903 849 leader_election.cc:343] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.099359 1442 leader_election.cc:290] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:58.099773 1302 raft_consensus.cc:2802] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:58.100965 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:58.101047 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d524089e6ab044ec961cfc9f587e8f86" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:58.100806 1302 raft_consensus.cc:695] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:58.101625 777 raft_consensus.cc:3058] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.102396 1534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap complete.
I20250625 13:51:58.106271 1302 consensus_queue.cc:237] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.106923 777 raft_consensus.cc:2466] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:58.107430 851 leader_election.cc:304] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:58.108330 1442 raft_consensus.cc:2802] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:58.108521 1442 raft_consensus.cc:695] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:58.108898 1442 consensus_queue.cc:237] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.110363 1165 ts_tablet_manager.cc:1918] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.110651 1165 log.cc:1199] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/2c1a98dc40684954a602567cf3476879
I20250625 13:51:58.111352 1165 ts_tablet_manager.cc:1939] T 2c1a98dc40684954a602567cf3476879 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:58.112264 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afa1c198dd6541809843244c7cc18194. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.113718 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 277) took 1762 ms. Trace:
I20250625 13:51:58.116952 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.116992 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d524089e6ab044ec961cfc9f587e8f86" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.118546 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413afd7b62d24fe8b9eceadd57e29fc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:58.118703 849 leader_election.cc:343] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.122324 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3249c5c2aa564027a2c594ca4c31e1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.123381 1167 log.cc:1199] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:58.123821 1167 ts_tablet_manager.cc:1939] T 09a8e15d57f14702ba0b5b52a94fc7c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:58.124137 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e7bc36f60d844f0d9d27755d6f58f63b" } modify_external_catalogs: true
I20250625 13:51:58.124219 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 242) took 2317 ms. Trace:
I20250625 13:51:58.124372 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e7bc36f60d844f0d9d27755d6f58f63b" } modify_external_catalogs: true
I20250625 13:51:58.124326 1167 rpcz_store.cc:276] 0625 13:51:55.807005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.808247 (+ 1242us) service_pool.cc:225] Handling call
0625 13:51:57.657926 (+1849679us) ts_tablet_manager.cc:1145] Deleting tablet 09a8e15d57f14702ba0b5b52a94fc7c4
0625 13:51:57.657941 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.989529 (+331588us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.062441 (+ 72912us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.124162 (+ 61721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1828703}
I20250625 13:51:58.124778 1167 tablet_replica.cc:331] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:58.124962 1167 raft_consensus.cc:2241] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.125145 1167 raft_consensus.cc:2270] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.114876 1165 rpcz_store.cc:276] 0625 13:51:56.351110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.351257 (+ 147us) service_pool.cc:225] Handling call
0625 13:51:57.618989 (+1267732us) ts_tablet_manager.cc:1145] Deleting tablet 2c1a98dc40684954a602567cf3476879
0625 13:51:57.618998 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:57.977430 (+358432us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.106083 (+128653us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.113589 (+ 7506us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4706,"tablet-delete.queue_time_us":1267536}
I20250625 13:51:58.125694 1167 ts_tablet_manager.cc:1905] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.126334 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:58.126925 1465 raft_consensus.cc:491] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.127063 1465 raft_consensus.cc:513] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.127701 1465 leader_election.cc:290] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.128082 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 278) took 1717 ms. Trace:
I20250625 13:51:58.127740 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a57268a18f4f00a39f8360f9c90737. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.128234 1165 rpcz_store.cc:276] 0625 13:51:56.410450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.410505 (+ 55us) service_pool.cc:225] Handling call
0625 13:51:58.127911 (+1717406us) ts_tablet_manager.cc:1145] Deleting tablet 01ba67b500b646d7aa55315f7ee23ed6
0625 13:51:58.127921 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.128031 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1717168}
I20250625 13:51:58.128996 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 279) took 1705 ms. Trace:
I20250625 13:51:58.129099 1165 rpcz_store.cc:276] 0625 13:51:56.423158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.423207 (+ 49us) service_pool.cc:225] Handling call
0625 13:51:58.128850 (+1705643us) ts_tablet_manager.cc:1145] Deleting tablet 991bb428def545fd979e9f1a80ce4d18
0625 13:51:58.128860 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.128956 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1705421}
W20250625 13:51:58.129191 985 leader_election.cc:343] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 67441795e22b445a90c88d76aa519ea5
I20250625 13:51:58.129302 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 2c1a98dc40684954a602567cf3476879 (table default.Table_1231c5f6367b4046beaa57696ea42ff4 [id=97fc76973f074d29903fa25a80ca534a]) successfully deleted
I20250625 13:51:58.129536 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 288) took 1567 ms. Trace:
I20250625 13:51:58.129663 1165 rpcz_store.cc:276] 0625 13:51:56.561680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.562847 (+ 1167us) service_pool.cc:225] Handling call
0625 13:51:58.129415 (+1566568us) ts_tablet_manager.cc:1145] Deleting tablet 3b0037d09796455cbaaadfaa9c010673
0625 13:51:58.129423 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.129507 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566312}
I20250625 13:51:58.132092 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 289) took 1564 ms. Trace:
I20250625 13:51:58.132735 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db6e9db2375144b38ec43ffca9ca4c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.133709 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67441795e22b445a90c88d76aa519ea5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:58.133985 916 raft_consensus.cc:2466] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:58.134315 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafc402d809245d189f3977162506ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.134421 983 leader_election.cc:304] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:58.134835 1465 raft_consensus.cc:2802] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:58.134945 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa52e79e00c4f9c97a05c353e11ae2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.134961 1465 raft_consensus.cc:491] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:58.135666 1465 raft_consensus.cc:3058] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.136474 1165 rpcz_store.cc:276] 0625 13:51:56.567638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.568057 (+ 419us) service_pool.cc:225] Handling call
0625 13:51:58.131948 (+1563891us) ts_tablet_manager.cc:1145] Deleting tablet bb1c7a85da844d34be805843bc60d350
0625 13:51:58.131957 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.132051 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563689}
I20250625 13:51:58.137046 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 290) took 1569 ms. Trace:
I20250625 13:51:58.137151 1165 rpcz_store.cc:276] 0625 13:51:56.568008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.570986 (+ 2978us) service_pool.cc:225] Handling call
0625 13:51:58.136922 (+1565936us) ts_tablet_manager.cc:1145] Deleting tablet 643928ca4fc94ce58431a0525f2abb99
0625 13:51:58.136931 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.137011 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565775}
I20250625 13:51:58.137533 1165 tablet_replica.cc:331] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:58.137811 1165 raft_consensus.cc:2241] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.137991 1165 raft_consensus.cc:2270] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.138721 1465 raft_consensus.cc:513] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.141006 1465 leader_election.cc:290] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.141324 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c81fab8e0b442e8d39556c1bce742e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:58.142117 985 leader_election.cc:343] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 67441795e22b445a90c88d76aa519ea5
W20250625 13:51:58.142367 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 01ba67b500b646d7aa55315f7ee23ed6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01ba67b500b646d7aa55315f7ee23ed6 already in progress: creating tablet
W20250625 13:51:58.142704 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 991bb428def545fd979e9f1a80ce4d18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 991bb428def545fd979e9f1a80ce4d18 already in progress: creating tablet
I20250625 13:51:58.142620 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67441795e22b445a90c88d76aa519ea5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
W20250625 13:51:58.142999 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3b0037d09796455cbaaadfaa9c010673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0037d09796455cbaaadfaa9c010673 already in progress: creating tablet
I20250625 13:51:58.143186 916 raft_consensus.cc:3058] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:58.143558 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bb1c7a85da844d34be805843bc60d350 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb1c7a85da844d34be805843bc60d350 already in progress: creating tablet
W20250625 13:51:58.143872 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 643928ca4fc94ce58431a0525f2abb99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 643928ca4fc94ce58431a0525f2abb99 already in progress: creating tablet
W20250625 13:51:58.145480 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed3c177bb6dd488aa54bee2a447016ff consensus state has no leader
I20250625 13:51:58.145948 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bc6c1303fba14b97b873dd23a423bf8c" } modify_external_catalogs: true
I20250625 13:51:58.146145 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bc6c1303fba14b97b873dd23a423bf8c" } modify_external_catalogs: true
I20250625 13:51:58.150511 916 raft_consensus.cc:2466] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:58.151178 983 leader_election.cc:304] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:58.150552 1534 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Replica starting. Triggering 9 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:58.151844 1465 raft_consensus.cc:2802] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:58.152225 1465 raft_consensus.cc:695] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:58.052947 877 tablet_service.cc:1468] Processing CreateTablet for tablet e2e6fbf3d81547f798b246a8b8f61387 (DEFAULT_TABLE table=default.Table_974313f23d974d75914dc9b2f7bd25d9 [id=7172d4663e37479e9998f617d88f8b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.152621 1465 consensus_queue.cc:237] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.157341 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 09a8e15d57f14702ba0b5b52a94fc7c4 (table default.Table_abfcbd249e4a4189aba6f16b5a66a0e7 [id=7d3d8e359ea945cc9495eb2454e1d85b]) successfully deleted
I20250625 13:51:58.159492 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 67441795e22b445a90c88d76aa519ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet abd6f9121a9d4b39adde1b880a875e23) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.163125 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.169499 1167 ts_tablet_manager.cc:1918] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:58.179291 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_78c7bbeb91584e54bef556a0af56c2a7" } modify_external_catalogs: true
I20250625 13:51:58.179569 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_78c7bbeb91584e54bef556a0af56c2a7" } modify_external_catalogs: true
W20250625 13:51:58.180184 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3c75deb64c74f0282efeed827659ec2 consensus state has no leader
I20250625 13:51:58.180613 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7e2035c1716744f7837845ba75f8b094"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.181272 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e2035c1716744f7837845ba75f8b094 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.179945 1555 tablet.cc:1722] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:51:58.186504 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e6fbf3d81547f798b246a8b8f61387. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.190639 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a57268a18f4f00a39f8360f9c90737 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.191211 896 tablet_service.cc:1468] Processing CreateTablet for tablet ef2c3757c0284aaca70ec317c05195d6 (DEFAULT_TABLE table=default.Table_44a75b53cab0470da32c4aedecdc2434 [id=7e8ccd792ea1452ba8c9fb719d11de52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.191696 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2c3757c0284aaca70ec317c05195d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.192430 1166 tablet_bootstrap.cc:492] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:58.192745 1166 ts_tablet_manager.cc:1397] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.575s user 0.007s sys 0.007s
I20250625 13:51:58.193861 1166 raft_consensus.cc:357] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.194247 1166 raft_consensus.cc:383] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.194384 1166 raft_consensus.cc:738] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.194752 1166 consensus_queue.cc:260] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.199043 890 tablet_service.cc:1515] Processing DeleteTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.199295 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 36da1da539e24f55978d516873cb0b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.151994 880 tablet_service.cc:1468] Processing CreateTablet for tablet 71cf3b95139245a4ba11b6bcaa3ab3f4 (DEFAULT_TABLE table=default.Table_ceec0bc74ee04dd5b21dcb6735342a42 [id=b3427651d7264e36922da55c8d267546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.200335 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71cf3b95139245a4ba11b6bcaa3ab3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.202699 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_22678b3f1a3b4dd4afcafdc967bf2e93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.203393 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22678b3f1a3b4dd4afcafdc967bf2e93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.205222 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2e709eb4a94545c58de6814fff368a89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.205739 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e709eb4a94545c58de6814fff368a89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.206539 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_254fead6128d4cb6b544757eb751f82b" } new_table_name: "default.Table_94aa82f5ff874e7b826fcb2abc61eeb7" modify_external_catalogs: true
I20250625 13:51:58.206878 1534 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fec9d8a79054a5991ae90d8a3101e32, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.202379 1165 ts_tablet_manager.cc:1905] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.220060 1165 ts_tablet_manager.cc:1918] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.220327 1165 log.cc:1199] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/14a12a5453e647b9946ca19e0ffb14ee
I20250625 13:51:58.220831 1165 ts_tablet_manager.cc:1939] T 14a12a5453e647b9946ca19e0ffb14ee P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:58.221694 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 301) took 1555 ms. Trace:
I20250625 13:51:58.221869 1165 rpcz_store.cc:276] 0625 13:51:56.665974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.672103 (+ 6129us) service_pool.cc:225] Handling call
0625 13:51:58.137464 (+1465361us) ts_tablet_manager.cc:1145] Deleting tablet 14a12a5453e647b9946ca19e0ffb14ee
0625 13:51:58.137475 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.202155 (+ 64680us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x611000ccf3f0. stack: 00007f4161830336 00007f41618300f0 00007f41608263ad 0000000000971004 0000000000970fcd 000000000097099a 00007f4174e5857d 00007f416e47426e 00007f416e5a24f7 00007f417107f137 00007f41710b11c4 00007f4171092c20 00007f41710929ad 0000000000a7573c 00007f41618759df 00007f4161878cab
0625 13:51:58.216629 (+ 14474us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.219937 (+ 3308us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.221633 (+ 1696us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146087296,"tablet-delete.queue_time_us":1457357}
I20250625 13:51:58.222456 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 14a12a5453e647b9946ca19e0ffb14ee (table default.Table_6cfe68ee62114f35be36b7f12d451a24 [id=09a0b05c11214fd0952325f08a002b8b]) successfully deleted
I20250625 13:51:58.222558 1165 tablet_replica.cc:331] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:58.222746 1165 raft_consensus.cc:2241] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.222949 1165 raft_consensus.cc:2270] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.223600 1165 ts_tablet_manager.cc:1905] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.226357 1166 ts_tablet_manager.cc:1428] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250625 13:51:58.227511 1166 tablet_bootstrap.cc:492] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:58.215047 1534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:58.242457 1166 tablet_bootstrap.cc:654] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:58.247603 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcf7fab53cae49908d2952cdaa07943a consensus state has no leader
I20250625 13:51:58.248162 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_817b97c6c8e742bf94f8c96a1485cf38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.248730 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_817b97c6c8e742bf94f8c96a1485cf38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.250633 1168 ts_tablet_manager.cc:1918] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:58.250773 1168 log.cc:1199] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/5786bb7716ef44689ef9862fc3ce17e9
I20250625 13:51:58.253168 592 catalog_manager.cc:5582] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20250625 13:51:58.253584 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d4f19ddc51324cc18242c6081f3f7de5 (table default.Table_08c36a3d2a03417982751548b18def03 [id=b8285b64bbee49f9be8741b252d03363]). Expected version 2 got 0
W20250625 13:51:58.254453 1550 log.cc:927] Time spent T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250625 13:51:58.257751 749 tablet_service.cc:1468] Processing CreateTablet for tablet e7bd4939714a4359846345ed0521701c (DEFAULT_TABLE table=default.Table_e78b8ab5d2584ab19f2560a2298b21cf [id=26deb489f6bb45f7b6b51d4a7f8e0c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.258157 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bd4939714a4359846345ed0521701c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.263623 756 tablet_service.cc:1468] Processing CreateTablet for tablet 535812d46d6444fda85d9767c6ca7e62 (DEFAULT_TABLE table=default.Table_e6f5787513b7437ab9fa137510881691 [id=26d6478fa3b646b99c7f2ee572a78be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.264009 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535812d46d6444fda85d9767c6ca7e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.267618 1545 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52994 (request call id 272) took 1577 ms. Trace:
I20250625 13:51:58.267678 752 tablet_service.cc:1468] Processing CreateTablet for tablet 7d51b886302149839efc6f547b944639 (DEFAULT_TABLE table=default.Table_e6f0bb3d8be34d36a5c48eeb803e5053 [id=2f3486b484e141fbad6554982346aea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.268080 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d51b886302149839efc6f547b944639. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.269477 1165 ts_tablet_manager.cc:1918] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.269598 1165 log.cc:1199] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/36da1da539e24f55978d516873cb0b27
I20250625 13:51:58.270171 1545 rpcz_store.cc:276] 0625 13:51:56.690084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.721149 (+ 31065us) service_pool.cc:225] Handling call
0625 13:51:58.267574 (+1546425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:51:56.923917 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0625 13:51:56.925405 (+ 1488us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0625 13:51:56.925416 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0625 13:51:56.925427 (+ 11us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0625 13:51:56.925498 (+ 71us) alter_schema_op.cc:128] START. Timestamp: 1750859516925442
0625 13:51:56.925507 (+ 9us) op_driver.cc:348] REPLICATION: starting
0625 13:51:56.956524 (+ 31017us) log.cc:844] Serialized 173 byte log entry
0625 13:51:57.867650 (+911126us) op_driver.cc:464] REPLICATION: finished
0625 13:51:57.980968 (+113318us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0625 13:51:57.981509 (+ 541us) log.cc:844] Serialized 5 byte log entry
0625 13:51:58.032250 (+ 50741us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.267484 (+235234us) log.cc:844] Serialized 14 byte log entry
0625 13:51:58.267531 (+ 47us) alter_schema_op.cc:73] Released schema lock
0625 13:51:58.267536 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":113239,"prepare.queue_time_us":491,"prepare.run_cpu_time_us":1175,"prepare.run_wall_time_us":32754,"raft.queue_time_us":4328,"raft.run_cpu_time_us":432,"raft.run_wall_time_us":165574,"replication_time_us":942107,"thread_start_us":118994,"threads_started":1}]]}
I20250625 13:51:58.271647 758 tablet_service.cc:1468] Processing CreateTablet for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 (DEFAULT_TABLE table=default.Table_7448453a4e8c44ab8ad28eef52b99d5e [id=fe62358f99bf4b89a66fdfdcd4670c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.272024 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efc8df3c8e24020b073a1b61bc9e0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.275669 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9be75d03121f4ea2906236ac3a5b0764 (DEFAULT_TABLE table=default.Table_9b8e7cde4c874918a309446de3d87d66 [id=8884bbb0de4d43bc89a4b9fc3d356c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.276068 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be75d03121f4ea2906236ac3a5b0764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.276147 1165 ts_tablet_manager.cc:1939] T 36da1da539e24f55978d516873cb0b27 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:58.276952 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 298) took 1611 ms. Trace:
I20250625 13:51:58.277084 1165 rpcz_store.cc:276] 0625 13:51:56.665205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.672035 (+ 6830us) service_pool.cc:225] Handling call
0625 13:51:58.222479 (+1550444us) ts_tablet_manager.cc:1145] Deleting tablet 36da1da539e24f55978d516873cb0b27
0625 13:51:58.222490 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.248891 (+ 26401us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.269333 (+ 20442us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.276908 (+ 7575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538968}
I20250625 13:51:58.277796 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 36da1da539e24f55978d516873cb0b27 (table default.Table_b079a0f225cf4ac98c2efc03bb74b889 [id=83a712e3adf74a95a5c2ea3fc5c0e14e]) successfully deleted
I20250625 13:51:58.279341 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:58.279726 747 tablet_service.cc:1468] Processing CreateTablet for tablet 4ae5f1d414794474b2178b431cbc1d44 (DEFAULT_TABLE table=default.Table_9234ef66a80b476b9614ca31733dfa74 [id=5b4f30f7fe414c589499632bdd0f8922]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.280241 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae5f1d414794474b2178b431cbc1d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.282347 1165 tablet_replica.cc:331] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:58.282574 1165 raft_consensus.cc:2241] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.282747 1165 raft_consensus.cc:2270] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.283239 755 tablet_service.cc:1468] Processing CreateTablet for tablet 5a289b9e3fac4fd8bacc22c235f7010c (DEFAULT_TABLE table=default.Table_a29eb5d240db4787866630bc39e35db5 [id=8e9a2d87890f4b5985c342a76d21eb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.283705 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a289b9e3fac4fd8bacc22c235f7010c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.286046 1302 raft_consensus.cc:491] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.286206 1302 raft_consensus.cc:513] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.286829 1302 leader_election.cc:290] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.287458 745 tablet_service.cc:1468] Processing CreateTablet for tablet 553af7c7326240838f37e3861d4e3357 (DEFAULT_TABLE table=default.Table_45ffe0341a354703ab5ab6c66bd27f18 [id=772bd28ed2964d3b9d09189c07cc34f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.287554 1165 ts_tablet_manager.cc:1905] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.287832 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553af7c7326240838f37e3861d4e3357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.290692 1171 tablet_bootstrap.cc:492] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.291200 1171 ts_tablet_manager.cc:1397] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.293s user 0.012s sys 0.018s
I20250625 13:51:58.292841 1170 ts_tablet_manager.cc:1428] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:51:58.295338 1171 raft_consensus.cc:357] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.296100 1171 raft_consensus.cc:383] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.296203 1171 raft_consensus.cc:738] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.296990 1171 consensus_queue.cc:260] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.297478 1171 ts_tablet_manager.cc:1428] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 13:51:58.298609 1171 tablet_bootstrap.cc:492] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.304747 1167 log.cc:1199] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/7108932fa8a241819ca90674a64b285e
I20250625 13:51:58.304267 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e281786e6e4a5d93b94b85b486655d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:58.306005 1171 tablet_bootstrap.cc:654] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:58.306715 711 leader_election.cc:343] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.307130 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e281786e6e4a5d93b94b85b486655d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.319726 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_715c56d22d6c4746a9983d7f8670d23e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.320398 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_715c56d22d6c4746a9983d7f8670d23e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.322163 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d43ba17ebfec4244894d1b6e23200c33
I20250625 13:51:58.322360 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9b8e7cde4c874918a309446de3d87d66" } modify_external_catalogs: true
I20250625 13:51:58.322540 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9b8e7cde4c874918a309446de3d87d66" } modify_external_catalogs: true
I20250625 13:51:58.322959 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1f2f5fdc47f0401bb672e1ae9c82473a" } new_table_name: "default.Table_b83d984846e24954a5702dc7392d94b1" modify_external_catalogs: true
W20250625 13:51:58.323877 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86af0ea6648045a1ac4ba903d20849b7 consensus state has no leader
I20250625 13:51:58.324934 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 535812d46d6444fda85d9767c6ca7e62
I20250625 13:51:58.326772 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1ca3f3760e6e41a29d004e6afe63ec98
I20250625 13:51:58.326998 1168 ts_tablet_manager.cc:1939] T 5786bb7716ef44689ef9862fc3ce17e9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:51:58.327518 711 leader_election.cc:343] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.327628 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7506cce0ec4c4ec68f25591cb98b1252
I20250625 13:51:58.327708 711 leader_election.cc:304] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:58.328296 1302 raft_consensus.cc:2747] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:58.328984 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 535812d46d6444fda85d9767c6ca7e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.329214 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.329432 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.336477 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 600d711975a74f66b9144758ad0dcfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.343405 1465 consensus_queue.cc:1035] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:58.344386 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 145) took 4088 ms. Trace:
I20250625 13:51:58.344529 1168 rpcz_store.cc:276] 0625 13:51:54.256194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.267990 (+ 11796us) service_pool.cc:225] Handling call
0625 13:51:58.057365 (+3789375us) ts_tablet_manager.cc:1145] Deleting tablet 5786bb7716ef44689ef9862fc3ce17e9
0625 13:51:58.057373 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.235283 (+177910us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.250500 (+ 15217us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.344339 (+ 93839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3789156}
I20250625 13:51:58.348935 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 146) took 4055 ms. Trace:
I20250625 13:51:58.349870 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4755ed4ecc244e44a38ad91f159a119b" } new_table_name: "default.Table_d18abaf5b02340fa82d3fee69699e1e3" modify_external_catalogs: true
I20250625 13:51:58.350485 1168 rpcz_store.cc:276] 0625 13:51:54.293190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.299386 (+ 6196us) service_pool.cc:225] Handling call
0625 13:51:58.348743 (+4049357us) ts_tablet_manager.cc:1145] Deleting tablet 8149aea0ff7646d3b94c9159f56e48ef
0625 13:51:58.348751 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.348894 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4048969}
I20250625 13:51:58.350934 1170 tablet_bootstrap.cc:492] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:58.351644 1167 ts_tablet_manager.cc:1939] T 7108932fa8a241819ca90674a64b285e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:58.352001 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 147) took 4048 ms. Trace:
I20250625 13:51:58.352025 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 267) took 2043 ms. Trace:
I20250625 13:51:58.352110 1168 rpcz_store.cc:276] 0625 13:51:54.303792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.303872 (+ 80us) service_pool.cc:225] Handling call
0625 13:51:58.351866 (+4047994us) ts_tablet_manager.cc:1145] Deleting tablet 93f449c80b604a6498f6fd1d1ba4770a
0625 13:51:58.351874 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.351972 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4047485}
W20250625 13:51:58.352628 711 consensus_peers.cc:487] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:58.358233 1168 tablet_replica.cc:331] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:58.358472 1167 rpcz_store.cc:276] 0625 13:51:56.308593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.308880 (+ 287us) service_pool.cc:225] Handling call
0625 13:51:58.124689 (+1815809us) ts_tablet_manager.cc:1145] Deleting tablet 7108932fa8a241819ca90674a64b285e
0625 13:51:58.124700 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.133206 (+ 8506us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.168676 (+ 35470us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.351990 (+183314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815600}
I20250625 13:51:58.358883 1168 raft_consensus.cc:2241] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.359074 1168 raft_consensus.cc:2270] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.359946 1167 tablet_replica.cc:331] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:51:58.359978 985 consensus_peers.cc:487] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a21335a06ec04da29b3351554c75f03c. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:58.360384 1167 raft_consensus.cc:2241] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.360584 1167 pending_rounds.cc:70] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:51:58.362362 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 7108932fa8a241819ca90674a64b285e (table default.Table_194007e5294849f9a6734e39b20d4fff [id=41a6138bc94a4a2790635b4ef0757c78]) successfully deleted
I20250625 13:51:58.364952 749 tablet_service.cc:1468] Processing CreateTablet for tablet 69c44d72516e4883aff636ca65929c11 (DEFAULT_TABLE table=default.Table_0d3b87078b494559b4a686a10a478402 [id=5a60593c92c040cfb7f22b8099e633a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.365442 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c44d72516e4883aff636ca65929c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.363360 895 tablet_service.cc:1468] Processing CreateTablet for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 (DEFAULT_TABLE table=default.Table_7448453a4e8c44ab8ad28eef52b99d5e [id=fe62358f99bf4b89a66fdfdcd4670c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.366639 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efc8df3c8e24020b073a1b61bc9e0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.363858 891 tablet_service.cc:1468] Processing CreateTablet for tablet 553af7c7326240838f37e3861d4e3357 (DEFAULT_TABLE table=default.Table_45ffe0341a354703ab5ab6c66bd27f18 [id=772bd28ed2964d3b9d09189c07cc34f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.364161 892 tablet_service.cc:1468] Processing CreateTablet for tablet 5a289b9e3fac4fd8bacc22c235f7010c (DEFAULT_TABLE table=default.Table_a29eb5d240db4787866630bc39e35db5 [id=8e9a2d87890f4b5985c342a76d21eb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.364388 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 535812d46d6444fda85d9767c6ca7e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.364568 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.364681 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d43ba17ebfec4244894d1b6e23200c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.367789 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553af7c7326240838f37e3861d4e3357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.371683 758 tablet_service.cc:1515] Processing DeleteTablet for tablet d43ba17ebfec4244894d1b6e23200c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.370280 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 4fa801f7d98e4794b6ca439638f29e69 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:58.374002 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 5786bb7716ef44689ef9862fc3ce17e9 (table default.Table_180b708af9884741956d0cb3898f9efd [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]) successfully deleted
I20250625 13:51:58.373951 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 535812d46d6444fda85d9767c6ca7e62 (DEFAULT_TABLE table=default.Table_e6f5787513b7437ab9fa137510881691 [id=26d6478fa3b646b99c7f2ee572a78be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.374362 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 7d51b886302149839efc6f547b944639 (DEFAULT_TABLE table=default.Table_e6f0bb3d8be34d36a5c48eeb803e5053 [id=2f3486b484e141fbad6554982346aea8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:58.374540 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8149aea0ff7646d3b94c9159f56e48ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8149aea0ff7646d3b94c9159f56e48ef already in progress: creating tablet
I20250625 13:51:58.374819 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a289b9e3fac4fd8bacc22c235f7010c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:58.374991 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 93f449c80b604a6498f6fd1d1ba4770a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93f449c80b604a6498f6fd1d1ba4770a already in progress: creating tablet
I20250625 13:51:58.379695 1564 tablet.cc:1722] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:51:58.382208 1510 raft_consensus.cc:491] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.382370 1510 raft_consensus.cc:513] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.382616 1167 pending_rounds.cc:77] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520579378835456 op_type: NO_OP noop_request { }
I20250625 13:51:58.383149 1510 leader_election.cc:290] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:51:58.384608 851 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 36da1da539e24f55978d516873cb0b27
I20250625 13:51:58.386185 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5ca183162fa545e7bd03a6bba000e7c9" } new_table_name: "default.Table_5d5a62a1608c45929c8fa1c853a06a27" modify_external_catalogs: true
I20250625 13:51:58.388521 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f4aa1ee5e69c4dd6a8917a9a9900768f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.389272 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4aa1ee5e69c4dd6a8917a9a9900768f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.364970 887 tablet_service.cc:1468] Processing CreateTablet for tablet 69c44d72516e4883aff636ca65929c11 (DEFAULT_TABLE table=default.Table_0d3b87078b494559b4a686a10a478402 [id=5a60593c92c040cfb7f22b8099e633a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.365265 882 tablet_service.cc:1468] Processing CreateTablet for tablet 9be75d03121f4ea2906236ac3a5b0764 (DEFAULT_TABLE table=default.Table_9b8e7cde4c874918a309446de3d87d66 [id=8884bbb0de4d43bc89a4b9fc3d356c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.365430 877 tablet_service.cc:1468] Processing CreateTablet for tablet 535812d46d6444fda85d9767c6ca7e62 (DEFAULT_TABLE table=default.Table_e6f5787513b7437ab9fa137510881691 [id=26d6478fa3b646b99c7f2ee572a78be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.365739 879 tablet_service.cc:1468] Processing CreateTablet for tablet 7d51b886302149839efc6f547b944639 (DEFAULT_TABLE table=default.Table_e6f0bb3d8be34d36a5c48eeb803e5053 [id=2f3486b484e141fbad6554982346aea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.365959 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.363806 883 tablet_service.cc:1468] Processing CreateTablet for tablet 4ae5f1d414794474b2178b431cbc1d44 (DEFAULT_TABLE table=default.Table_9234ef66a80b476b9614ca31733dfa74 [id=5b4f30f7fe414c589499632bdd0f8922]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.390105 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae5f1d414794474b2178b431cbc1d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.391171 880 tablet_service.cc:1468] Processing CreateTablet for tablet e7bd4939714a4359846345ed0521701c (DEFAULT_TABLE table=default.Table_e78b8ab5d2584ab19f2560a2298b21cf [id=26deb489f6bb45f7b6b51d4a7f8e0c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.392096 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:58.392971 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bd4939714a4359846345ed0521701c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.394539 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:58.394603 1171 tablet_bootstrap.cc:492] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.394999 1171 ts_tablet_manager.cc:1397] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.097s user 0.013s sys 0.019s
I20250625 13:51:58.396684 1171 raft_consensus.cc:357] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.397060 1171 raft_consensus.cc:383] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.397269 1171 raft_consensus.cc:738] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.397745 1171 consensus_queue.cc:260] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.398543 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c44d72516e4883aff636ca65929c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.404590 1171 ts_tablet_manager.cc:1428] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 13:51:58.405177 1171 tablet_bootstrap.cc:492] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.405375 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be75d03121f4ea2906236ac3a5b0764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.407047 1545 tablet.cc:1717] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Skipping requested alter to schema version 3, tablet already version 3
I20250625 13:51:58.410468 1171 tablet_bootstrap.cc:654] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.412772 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d51b886302149839efc6f547b944639. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.413802 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d51b886302149839efc6f547b944639. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.414163 1170 tablet_bootstrap.cc:654] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.382778 1014 tablet_service.cc:1468] Processing CreateTablet for tablet e7bd4939714a4359846345ed0521701c (DEFAULT_TABLE table=default.Table_e78b8ab5d2584ab19f2560a2298b21cf [id=26deb489f6bb45f7b6b51d4a7f8e0c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.417191 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 5a289b9e3fac4fd8bacc22c235f7010c (DEFAULT_TABLE table=default.Table_a29eb5d240db4787866630bc39e35db5 [id=8e9a2d87890f4b5985c342a76d21eb5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.418682 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535812d46d6444fda85d9767c6ca7e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.419451 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 553af7c7326240838f37e3861d4e3357 (DEFAULT_TABLE table=default.Table_45ffe0341a354703ab5ab6c66bd27f18 [id=772bd28ed2964d3b9d09189c07cc34f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.421466 1167 raft_consensus.cc:2887] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:58.422097 1168 ts_tablet_manager.cc:1905] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.423846 1171 tablet_bootstrap.cc:492] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.423853 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 4ae5f1d414794474b2178b431cbc1d44 (DEFAULT_TABLE table=default.Table_9234ef66a80b476b9614ca31733dfa74 [id=5b4f30f7fe414c589499632bdd0f8922]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:58.424721 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 86af0ea6648045a1ac4ba903d20849b7 consensus state has no leader
I20250625 13:51:58.424816 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d43ba17ebfec4244894d1b6e23200c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.425709 1171 ts_tablet_manager.cc:1397] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20250625 13:51:58.425763 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 535812d46d6444fda85d9767c6ca7e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.426414 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 (DEFAULT_TABLE table=default.Table_7448453a4e8c44ab8ad28eef52b99d5e [id=fe62358f99bf4b89a66fdfdcd4670c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.426823 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36da1da539e24f55978d516873cb0b27" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.427297 1171 raft_consensus.cc:357] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:58.427675 849 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.427812 849 leader_election.cc:304] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:58.430164 1022 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42238 (request call id 344) took 1047 ms. Trace:
I20250625 13:51:58.430253 1022 rpcz_store.cc:276] 0625 13:51:57.380447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.782414 (+401967us) service_pool.cc:225] Handling call
0625 13:51:57.796243 (+ 13829us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0625 13:51:57.796405 (+ 162us) ts_tablet_manager.cc:676] Creating new metadata...
0625 13:51:58.170117 (+373712us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.427910 (+257793us) tablet_replica.cc:206] Creating consensus instance
0625 13:51:58.428065 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1011584}
I20250625 13:51:58.429917 1030 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42238 (request call id 343) took 1001 ms. Trace:
I20250625 13:51:58.432443 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9be75d03121f4ea2906236ac3a5b0764 (DEFAULT_TABLE table=default.Table_9b8e7cde4c874918a309446de3d87d66 [id=8884bbb0de4d43bc89a4b9fc3d356c4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.436726 1167 raft_consensus.cc:2270] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.436743 1534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:51:58.438246 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.439421 1166 tablet_bootstrap.cc:492] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:58.439527 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 69c44d72516e4883aff636ca65929c11 (DEFAULT_TABLE table=default.Table_0d3b87078b494559b4a686a10a478402 [id=5a60593c92c040cfb7f22b8099e633a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.440284 1167 ts_tablet_manager.cc:1905] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.442020 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bd4939714a4359846345ed0521701c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.440788 1030 rpcz_store.cc:276] 0625 13:51:57.380326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.774039 (+393713us) service_pool.cc:225] Handling call
0625 13:51:57.793784 (+ 19745us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0625 13:51:57.794504 (+ 720us) ts_tablet_manager.cc:676] Creating new metadata...
0625 13:51:58.174374 (+379870us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.373547 (+199173us) tablet_replica.cc:206] Creating consensus instance
0625 13:51:58.381547 (+ 8000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:51:58.446523 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e22c886829ef45be814505c0f1705376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.447208 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e22c886829ef45be814505c0f1705376 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.447999 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535812d46d6444fda85d9767c6ca7e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.448228 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f2851488b3864c7b893b63c84369288d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.448849 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2851488b3864c7b893b63c84369288d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.449996 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0376b3d69a954c6881b4fd9d11e4b86b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.450492 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0376b3d69a954c6881b4fd9d11e4b86b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.450644 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_af0069799827448399dea82497b0b4d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.451118 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af0069799827448399dea82497b0b4d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.452039 1166 ts_tablet_manager.cc:1397] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.225s user 0.010s sys 0.000s
I20250625 13:51:58.454867 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a289b9e3fac4fd8bacc22c235f7010c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.457607 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7efc8df3c8e24020b073a1b61bc9e0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.459059 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553af7c7326240838f37e3861d4e3357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.459488 1564 tablet.cc:1717] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Skipping requested alter to schema version 3, tablet already version 3
I20250625 13:51:58.460495 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae5f1d414794474b2178b431cbc1d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.461683 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be75d03121f4ea2906236ac3a5b0764. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:58.464401 711 consensus_peers.cc:487] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:58.459023 1166 raft_consensus.cc:357] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.464563 1166 raft_consensus.cc:383] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.464700 1166 raft_consensus.cc:738] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.464980 1166 consensus_queue.cc:260] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.468732 1166 ts_tablet_manager.cc:1428] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 13:51:58.469076 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c44d72516e4883aff636ca65929c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.472226 1166 tablet_bootstrap.cc:492] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:58.476996 1166 tablet_bootstrap.cc:654] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.477308 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.479688 1165 ts_tablet_manager.cc:1918] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.479905 1165 log.cc:1199] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/389d81329b8d430a9098b0630612fdcb
I20250625 13:51:58.480398 1165 ts_tablet_manager.cc:1939] T 389d81329b8d430a9098b0630612fdcb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:58.487449 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 299) took 1821 ms. Trace:
I20250625 13:51:58.487903 1165 rpcz_store.cc:276] 0625 13:51:56.665486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.672061 (+ 6575us) service_pool.cc:225] Handling call
0625 13:51:58.282274 (+1610213us) ts_tablet_manager.cc:1145] Deleting tablet 389d81329b8d430a9098b0630612fdcb
0625 13:51:58.282284 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.291244 (+ 8960us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.479538 (+188294us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.487376 (+ 7838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598585}
I20250625 13:51:58.489020 1166 tablet_bootstrap.cc:492] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:58.488956 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 389d81329b8d430a9098b0630612fdcb (table default.Table_1a06c7c260544cce91ab86dad2d62346 [id=aff513ce6735496784852aef3728d643]) successfully deleted
I20250625 13:51:58.489284 1166 ts_tablet_manager.cc:1397] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20250625 13:51:58.489524 1165 tablet_replica.cc:331] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:58.490126 1165 raft_consensus.cc:2241] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.490792 1165 raft_consensus.cc:2270] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.490175 1166 raft_consensus.cc:357] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.491504 1166 raft_consensus.cc:383] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.491638 1166 raft_consensus.cc:738] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.491828 1165 ts_tablet_manager.cc:1905] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.491956 1166 consensus_queue.cc:260] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.496413 1166 ts_tablet_manager.cc:1428] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:51:58.498543 1166 tablet_bootstrap.cc:492] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:58.500784 1373 raft_consensus.cc:491] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.500921 1373 raft_consensus.cc:513] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.501531 1373 leader_election.cc:290] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.502940 1166 tablet_bootstrap.cc:654] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.508591 1165 ts_tablet_manager.cc:1918] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.512519 1165 log.cc:1199] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/b7641869a0bb463698244ff445d87a1f
I20250625 13:51:58.513432 1165 ts_tablet_manager.cc:1939] T b7641869a0bb463698244ff445d87a1f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:58.514869 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 300) took 1849 ms. Trace:
I20250625 13:51:58.515995 1512 raft_consensus.cc:2747] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:58.516758 1165 rpcz_store.cc:276] 0625 13:51:56.665730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.672084 (+ 6354us) service_pool.cc:225] Handling call
0625 13:51:58.489405 (+1817321us) ts_tablet_manager.cc:1145] Deleting tablet b7641869a0bb463698244ff445d87a1f
0625 13:51:58.489419 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.497047 (+ 7628us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.508436 (+ 11389us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.514809 (+ 6373us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803780}
I20250625 13:51:58.517201 1569 tablet.cc:1722] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:51:58.521437 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41a7862601f04fd7a938f7e5d5a4c0f9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.522222 916 raft_consensus.cc:1273] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Refusing update from remote peer 5c1aa07ab6bc4b0089e46951240a37c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:51:58.522558 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41a7862601f04fd7a938f7e5d5a4c0f9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:58.523242 1510 raft_consensus.cc:491] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c1aa07ab6bc4b0089e46951240a37c9)
I20250625 13:51:58.527410 1510 raft_consensus.cc:513] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.528165 1510 leader_election.cc:290] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:58.531580 1171 raft_consensus.cc:383] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:51:58.532446 849 leader_election.cc:343] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 76f074156559406c875e65796e6b5ee6
W20250625 13:51:58.532668 851 leader_election.cc:343] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 76f074156559406c875e65796e6b5ee6
I20250625 13:51:58.533035 851 leader_election.cc:304] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:58.534724 1510 raft_consensus.cc:2747] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:51:58.546525 1171 raft_consensus.cc:738] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.548979 1171 consensus_queue.cc:260] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.549520 1171 ts_tablet_manager.cc:1428] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.124s user 0.002s sys 0.000s
I20250625 13:51:58.550460 1171 tablet_bootstrap.cc:492] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.555493 1171 tablet_bootstrap.cc:654] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.573216 1167 ts_tablet_manager.cc:1918] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:58.580416 1170 tablet_bootstrap.cc:492] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:58.580615 1170 ts_tablet_manager.cc:1397] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.235s user 0.004s sys 0.002s
I20250625 13:51:58.581326 1170 raft_consensus.cc:357] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.581555 1170 raft_consensus.cc:383] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.581636 1170 raft_consensus.cc:738] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.581930 1170 consensus_queue.cc:260] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.582417 1170 ts_tablet_manager.cc:1428] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250625 13:51:58.590173 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af05fba6fe2042aab5dd47b30e4dfcf3 consensus state has no leader
W20250625 13:51:58.592103 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abd6f9121a9d4b39adde1b880a875e23 consensus state has no leader
W20250625 13:51:58.592470 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae4a105e80c1460cbb21e1bdb9b356b1 consensus state has no leader
I20250625 13:51:58.592727 591 master_service.cc:948] ReplaceTablet: received request to replace tablet d524089e6ab044ec961cfc9f587e8f86 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:58.593359 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 0b129117db064a15b27e82439637cf22 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:58.593909 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ee7cfdccf06f4d94982b3b66c9bf1a1d" } new_table_name: "default.Table_05bee7b38e4a49e99da18772c39769d7" modify_external_catalogs: true
I20250625 13:51:58.595039 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 15e58fbec94e4ff9b7ac9830b576fe7f from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:58.595372 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4145b0b039bc4d73a22b72a6340b02c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:58.595665 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_513b4c6ebe2942e1a4018f3d8e385853" } new_table_name: "default.Table_1e0aeab3aac8463aad52e947a2aadb14" modify_external_catalogs: true
W20250625 13:51:58.595909 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4145b0b039bc4d73a22b72a6340b02c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.596139 1168 ts_tablet_manager.cc:1918] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.596271 1168 log.cc:1199] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/e75b5fb51373451a94cb5ca59cdcefac
I20250625 13:51:58.597518 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_df6a42eba40144cf94d09e54694a85b9" } modify_external_catalogs: true
I20250625 13:51:58.597707 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_df6a42eba40144cf94d09e54694a85b9" } modify_external_catalogs: true
I20250625 13:51:58.598264 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9be75d03121f4ea2906236ac3a5b0764
I20250625 13:51:58.598912 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_528b80e737c9462194cd9b50d91b7db9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:58.599149 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 9be75d03121f4ea2906236ac3a5b0764 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:51:58.599403 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_528b80e737c9462194cd9b50d91b7db9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.599483 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 9be75d03121f4ea2906236ac3a5b0764 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.601092 1510 raft_consensus.cc:491] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.601248 1510 raft_consensus.cc:513] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.601819 1510 leader_election.cc:290] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:58.602962 1171 tablet_bootstrap.cc:492] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.603085 1171 ts_tablet_manager.cc:1397] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.002s
I20250625 13:51:58.603869 1171 raft_consensus.cc:357] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.603920 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_248a6183f6d44e43ab1fa4722ec9a52e" } modify_external_catalogs: true
I20250625 13:51:58.604023 1171 raft_consensus.cc:383] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.604058 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_248a6183f6d44e43ab1fa4722ec9a52e" } modify_external_catalogs: true
I20250625 13:51:58.604116 1171 raft_consensus.cc:738] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.604351 1171 consensus_queue.cc:260] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.604686 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_866deeb74a8743f0a579513bf580c9e9" } modify_external_catalogs: true
I20250625 13:51:58.604839 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_866deeb74a8743f0a579513bf580c9e9" } modify_external_catalogs: true
I20250625 13:51:58.605057 1171 ts_tablet_manager.cc:1428] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:58.605326 1171 tablet_bootstrap.cc:492] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.607046 1171 tablet_bootstrap.cc:654] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.608142 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0e7f3b83e2c7403eb46bc273da03cee8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.608649 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e7f3b83e2c7403eb46bc273da03cee8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.612049 1168 ts_tablet_manager.cc:1939] T e75b5fb51373451a94cb5ca59cdcefac P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:58.517509 1166 tablet_bootstrap.cc:492] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:58.612583 1166 ts_tablet_manager.cc:1397] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.003s
I20250625 13:51:58.612717 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet b7641869a0bb463698244ff445d87a1f (table default.Table_6e5d1377cfc84651afd53a78735468d8 [id=bd476a97393841899b4cdb426c39e6e9]) successfully deleted
I20250625 13:51:58.613045 1165 tablet_replica.cc:331] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:58.613468 1166 raft_consensus.cc:357] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.613663 1166 raft_consensus.cc:383] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.613754 1166 raft_consensus.cc:738] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.613942 1165 raft_consensus.cc:2241] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:58.613977 1166 consensus_queue.cc:260] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.614554 1166 ts_tablet_manager.cc:1428] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:51:58.614849 1166 tablet_bootstrap.cc:492] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:58.615371 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:58.616693 1166 tablet_bootstrap.cc:654] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:58.522589 711 leader_election.cc:343] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:58.617681 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 95082us
W20250625 13:51:58.618167 711 leader_election.cc:343] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.618300 711 leader_election.cc:304] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:58.618870 1302 consensus_queue.cc:1035] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:58.619328 1302 raft_consensus.cc:2747] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:58.619539 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8149aea0ff7646d3b94c9159f56e48ef" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:58.620965 849 leader_election.cc:343] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.602586 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8149aea0ff7646d3b94c9159f56e48ef" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:58.621448 777 raft_consensus.cc:2466] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:58.610723 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 9be75d03121f4ea2906236ac3a5b0764 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.621848 1170 tablet_bootstrap.cc:492] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:58.622262 851 leader_election.cc:304] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:58.622771 1510 raft_consensus.cc:2802] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:58.622871 1510 raft_consensus.cc:491] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:58.622989 1510 raft_consensus.cc:3058] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.623553 1170 tablet_bootstrap.cc:654] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.626045 1510 raft_consensus.cc:513] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.627255 1510 leader_election.cc:290] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:58.627628 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8149aea0ff7646d3b94c9159f56e48ef" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:58.627740 1165 pending_rounds.cc:70] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:58.627837 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8149aea0ff7646d3b94c9159f56e48ef" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:58.628012 777 raft_consensus.cc:3058] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:58.628096 849 leader_election.cc:343] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.628854 1167 log.cc:1199] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/5786bb7716ef44689ef9862fc3ce17e9
I20250625 13:51:58.629698 1167 ts_tablet_manager.cc:1939] T 5786bb7716ef44689ef9862fc3ce17e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:58.626040 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 148) took 4293 ms. Trace:
I20250625 13:51:58.630034 1168 rpcz_store.cc:276] 0625 13:51:54.332397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.332449 (+ 52us) service_pool.cc:225] Handling call
0625 13:51:58.358147 (+4025698us) ts_tablet_manager.cc:1145] Deleting tablet e75b5fb51373451a94cb5ca59cdcefac
0625 13:51:58.358157 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.566256 (+208099us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.596002 (+ 29746us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.625970 (+ 29968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4004864}
I20250625 13:51:58.630466 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 165) took 4072 ms. Trace:
I20250625 13:51:58.630478 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 268) took 2321 ms. Trace:
I20250625 13:51:58.630543 1168 rpcz_store.cc:276] 0625 13:51:54.558305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.558400 (+ 95us) service_pool.cc:225] Handling call
0625 13:51:58.630320 (+4071920us) ts_tablet_manager.cc:1145] Deleting tablet 19017cc372af43b2b5074a8990375ede
0625 13:51:58.630329 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.630420 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4060096}
I20250625 13:51:58.630617 1167 rpcz_store.cc:276] 0625 13:51:56.308734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.309121 (+ 387us) service_pool.cc:225] Handling call
0625 13:51:58.358898 (+2049777us) ts_tablet_manager.cc:1145] Deleting tablet 5786bb7716ef44689ef9862fc3ce17e9
0625 13:51:58.358912 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.489693 (+130781us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.573074 (+ 83381us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.630431 (+ 57357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049560}
I20250625 13:51:58.630908 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 167) took 4060 ms. Trace:
I20250625 13:51:58.631042 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 5786bb7716ef44689ef9862fc3ce17e9 (table default.Table_180b708af9884741956d0cb3898f9efd [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]) successfully deleted
I20250625 13:51:58.630983 1168 rpcz_store.cc:276] 0625 13:51:54.570849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.570947 (+ 98us) service_pool.cc:225] Handling call
0625 13:51:58.630764 (+4059817us) ts_tablet_manager.cc:1145] Deleting tablet f033bfa4925b4d1c971789eb77efd92c
0625 13:51:58.630772 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.630860 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4056120}
I20250625 13:51:58.631363 1167 tablet_replica.cc:331] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:58.631352 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 166) took 4062 ms. Trace:
I20250625 13:51:58.631433 1168 rpcz_store.cc:276] 0625 13:51:54.568505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.568648 (+ 143us) service_pool.cc:225] Handling call
0625 13:51:58.631196 (+4062548us) ts_tablet_manager.cc:1145] Deleting tablet c911d9ecbded417daf70e85c7cbcd52e
0625 13:51:58.631204 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.631288 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4056355}
I20250625 13:51:58.631588 1167 raft_consensus.cc:2241] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.631700 1168 tablet_replica.cc:331] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:58.631742 1167 raft_consensus.cc:2270] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.631893 1168 raft_consensus.cc:2241] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.632103 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet e75b5fb51373451a94cb5ca59cdcefac (table default.Table_e75519f3b879406ea7bb8f515825f274 [id=ffb244ee24124e8a83db14b5747abed7]) successfully deleted
I20250625 13:51:58.632292 1167 ts_tablet_manager.cc:1905] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:58.632344 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 19017cc372af43b2b5074a8990375ede because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19017cc372af43b2b5074a8990375ede already in progress: creating tablet
W20250625 13:51:58.632547 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet f033bfa4925b4d1c971789eb77efd92c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f033bfa4925b4d1c971789eb77efd92c already in progress: creating tablet
W20250625 13:51:58.632794 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c911d9ecbded417daf70e85c7cbcd52e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c911d9ecbded417daf70e85c7cbcd52e already in progress: creating tablet
I20250625 13:51:58.640430 1170 tablet_bootstrap.cc:492] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:58.640628 1170 ts_tablet_manager.cc:1397] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250625 13:51:58.641459 1170 raft_consensus.cc:357] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.641664 1170 raft_consensus.cc:383] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.641775 1170 raft_consensus.cc:738] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.642089 1170 consensus_queue.cc:260] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.643131 1170 ts_tablet_manager.cc:1428] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:58.643443 1170 tablet_bootstrap.cc:492] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:58.644230 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:58.645394 1166 tablet_bootstrap.cc:492] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:58.645551 1166 ts_tablet_manager.cc:1397] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.001s
I20250625 13:51:58.645866 1171 tablet_bootstrap.cc:492] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.646360 1166 raft_consensus.cc:357] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.646605 1166 raft_consensus.cc:383] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.646687 1166 raft_consensus.cc:738] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.646761 777 raft_consensus.cc:2466] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:58.646937 1166 consensus_queue.cc:260] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.647161 1171 ts_tablet_manager.cc:1397] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20250625 13:51:58.647441 1166 ts_tablet_manager.cc:1428] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:58.647468 851 leader_election.cc:304] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:58.647725 1166 tablet_bootstrap.cc:492] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:58.648061 1510 raft_consensus.cc:2802] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:58.648221 1510 raft_consensus.cc:695] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:58.648525 1510 consensus_queue.cc:237] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.648976 1171 raft_consensus.cc:357] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.649173 1171 raft_consensus.cc:383] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.649257 1171 raft_consensus.cc:738] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.649405 1166 tablet_bootstrap.cc:654] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.649513 1171 consensus_queue.cc:260] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.649979 1171 ts_tablet_manager.cc:1428] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 13:51:58.650279 1171 tablet_bootstrap.cc:492] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.656555 1170 tablet_bootstrap.cc:654] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.660159 1564 tablet.cc:1722] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:51:58.661852 1166 tablet_bootstrap.cc:492] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:58.662038 1166 ts_tablet_manager.cc:1397] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250625 13:51:58.663518 1171 tablet_bootstrap.cc:654] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.663429 1166 raft_consensus.cc:357] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.663594 1166 raft_consensus.cc:383] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.663673 1166 raft_consensus.cc:738] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.647487 1168 raft_consensus.cc:2270] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.663892 1166 consensus_queue.cc:260] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.664269 1165 pending_rounds.cc:77] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520583504060416 op_type: NO_OP noop_request { }
I20250625 13:51:58.664381 1165 raft_consensus.cc:2887] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:58.664480 1166 ts_tablet_manager.cc:1428] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:58.664522 1165 raft_consensus.cc:2270] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.664767 1166 tablet_bootstrap.cc:492] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:58.664929 1545 tablet.cc:1722] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:51:58.666350 1166 tablet_bootstrap.cc:654] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.668654 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:51:58.671278 1170 tablet_bootstrap.cc:492] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:58.671486 1170 ts_tablet_manager.cc:1397] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20250625 13:51:58.672223 1170 raft_consensus.cc:357] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.672425 1170 raft_consensus.cc:383] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.674049 1168 ts_tablet_manager.cc:1905] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.679347 1170 raft_consensus.cc:738] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.680313 1170 consensus_queue.cc:260] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.680879 1170 ts_tablet_manager.cc:1428] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250625 13:51:58.681206 1170 tablet_bootstrap.cc:492] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:58.684055 1167 ts_tablet_manager.cc:1918] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.684202 1167 log.cc:1199] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/13620f6ca65342b4ad7f1ba4b437bde3
I20250625 13:51:58.684561 1167 ts_tablet_manager.cc:1939] T 13620f6ca65342b4ad7f1ba4b437bde3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:58.684897 1170 tablet_bootstrap.cc:654] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.687883 1171 tablet_bootstrap.cc:492] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.688045 1171 ts_tablet_manager.cc:1397] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.004s
I20250625 13:51:58.688657 1165 ts_tablet_manager.cc:1905] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.688804 1171 raft_consensus.cc:357] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.689011 1171 raft_consensus.cc:383] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.689110 1171 raft_consensus.cc:738] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.685833 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 269) took 2376 ms. Trace:
I20250625 13:51:58.689369 1171 consensus_queue.cc:260] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.691251 1167 rpcz_store.cc:276] 0625 13:51:56.308836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.309361 (+ 525us) service_pool.cc:225] Handling call
0625 13:51:58.631277 (+2321916us) ts_tablet_manager.cc:1145] Deleting tablet 13620f6ca65342b4ad7f1ba4b437bde3
0625 13:51:58.631288 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.652730 (+ 21442us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.683931 (+ 31201us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.685763 (+ 1832us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2321762}
I20250625 13:51:58.692502 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 13620f6ca65342b4ad7f1ba4b437bde3 (table default.Table_6199817767f74031b8a500d21174d5a6 [id=bbf70badd2ea45ae88881afe31a3de40]) successfully deleted
I20250625 13:51:58.693346 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 270) took 2373 ms. Trace:
I20250625 13:51:58.694339 1167 rpcz_store.cc:276] 0625 13:51:56.320138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.320791 (+ 653us) service_pool.cc:225] Handling call
0625 13:51:58.693187 (+2372396us) ts_tablet_manager.cc:1145] Deleting tablet 1f6be7189f8f45ccb329ef5d2fd8da8d
0625 13:51:58.693197 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.693296 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2372181}
W20250625 13:51:58.695015 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f6be7189f8f45ccb329ef5d2fd8da8d already in progress: creating tablet
I20250625 13:51:58.696543 1170 tablet_bootstrap.cc:492] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:58.696904 1170 ts_tablet_manager.cc:1397] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20250625 13:51:58.692488 1171 ts_tablet_manager.cc:1428] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 13:51:58.692705 1510 raft_consensus.cc:491] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.696660 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 272) took 2366 ms. Trace:
I20250625 13:51:58.697337 1510 raft_consensus.cc:513] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.697356 1167 rpcz_store.cc:276] 0625 13:51:56.330139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.330196 (+ 57us) service_pool.cc:225] Handling call
0625 13:51:58.695432 (+2365236us) ts_tablet_manager.cc:1145] Deleting tablet bd5ba76c130a4711af5b21e9eb8562e7
0625 13:51:58.695440 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.696316 (+ 876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2365005}
W20250625 13:51:58.697861 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bd5ba76c130a4711af5b21e9eb8562e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5ba76c130a4711af5b21e9eb8562e7 already in progress: creating tablet
I20250625 13:51:58.698007 1510 leader_election.cc:290] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:58.698520 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "600d711975a74f66b9144758ad0dcfb0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.698750 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "600d711975a74f66b9144758ad0dcfb0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:58.699012 777 raft_consensus.cc:2466] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:58.699137 1167 tablet_replica.cc:331] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:58.699548 1167 raft_consensus.cc:2241] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.699551 851 leader_election.cc:304] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:51:58.699867 849 leader_election.cc:343] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.700747 1167 raft_consensus.cc:2270] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.701102 1171 tablet_bootstrap.cc:492] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.701228 1510 raft_consensus.cc:2802] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:58.701325 1510 raft_consensus.cc:491] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:58.701409 1510 raft_consensus.cc:3058] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.700276 1170 raft_consensus.cc:357] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.701758 1170 raft_consensus.cc:383] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.701867 1170 raft_consensus.cc:738] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.702301 1167 ts_tablet_manager.cc:1905] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.703982 1171 tablet_bootstrap.cc:654] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.704849 1170 consensus_queue.cc:260] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.705310 1170 ts_tablet_manager.cc:1428] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250625 13:51:58.705585 1170 tablet_bootstrap.cc:492] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:58.707144 1170 tablet_bootstrap.cc:654] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.709993 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_675c4ec332bd4560b7e40c2bc7ce5541 [id=a64872263d0d4aeaaac5a6df29cb73bd] alter complete (version 3)
I20250625 13:51:58.704452 1510 raft_consensus.cc:513] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.715236 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "600d711975a74f66b9144758ad0dcfb0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:58.715531 777 raft_consensus.cc:3058] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.715929 1510 leader_election.cc:290] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:58.715993 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_91567a207fcd4540937c0e047ea86de5" } modify_external_catalogs: true
I20250625 13:51:58.716176 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_91567a207fcd4540937c0e047ea86de5" } modify_external_catalogs: true
I20250625 13:51:58.717303 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b549bc8cead4af4a72f41f51655602c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:58.717502 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c3f0ca6fa5482db7f57143911c838f consensus state has no leader
W20250625 13:51:58.719228 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3c75deb64c74f0282efeed827659ec2 consensus state has no leader
I20250625 13:51:58.719666 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9150143ab7394f1baf2fb087e8265dfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.720445 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9150143ab7394f1baf2fb087e8265dfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.720677 777 raft_consensus.cc:2466] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:58.721282 851 leader_election.cc:304] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:58.722350 596 master_service.cc:948] ReplaceTablet: received request to replace tablet a30a35e3639a43c5b078169060824723 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:58.722972 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 32a57268a18f4f00a39f8360f9c90737 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:58.723531 1510 raft_consensus.cc:2802] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:58.723994 1510 raft_consensus.cc:695] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:58.724886 1510 consensus_queue.cc:237] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.734009 749 tablet_service.cc:1468] Processing CreateTablet for tablet 663588b7e9ee430faa180fec00897e22 (DEFAULT_TABLE table=default.Table_7e2035c1716744f7837845ba75f8b094 [id=4570d94b0b6f4e5db7ba8098dab81503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.734504 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663588b7e9ee430faa180fec00897e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.734475 880 tablet_service.cc:1468] Processing CreateTablet for tablet cab4329d58404014833ae5029f8f296d (DEFAULT_TABLE table=default.Table_3947dfd6b9454b7eb988acee97d6e311 [id=e7367d6c3e05483cbff3ec85064df30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.734970 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab4329d58404014833ae5029f8f296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.735196 879 tablet_service.cc:1468] Processing CreateTablet for tablet 0a90fd8cb29a4fffa65129dbca9ad84c (DEFAULT_TABLE table=default.Table_d568530b5b6e472fb3c3fd4b301a6dd2 [id=0f41315f3f4e46edb8f4b0f8a6b0f25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.735635 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a90fd8cb29a4fffa65129dbca9ad84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.738313 877 tablet_service.cc:1468] Processing CreateTablet for tablet 663588b7e9ee430faa180fec00897e22 (DEFAULT_TABLE table=default.Table_7e2035c1716744f7837845ba75f8b094 [id=4570d94b0b6f4e5db7ba8098dab81503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.738317 758 tablet_service.cc:1468] Processing CreateTablet for tablet 0a90fd8cb29a4fffa65129dbca9ad84c (DEFAULT_TABLE table=default.Table_d568530b5b6e472fb3c3fd4b301a6dd2 [id=0f41315f3f4e46edb8f4b0f8a6b0f25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.738672 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663588b7e9ee430faa180fec00897e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.738682 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a90fd8cb29a4fffa65129dbca9ad84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.739606 882 tablet_service.cc:1468] Processing CreateTablet for tablet 189a9347cd5d4b5f955e5c0706aa3947 (DEFAULT_TABLE table=default.Table_a4182ebe832a4ef2a6b0a736bfcbc4b4 [id=b4a114b3b86a4c6ba3a8d3b69af100d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.739996 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189a9347cd5d4b5f955e5c0706aa3947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.743703 745 tablet_service.cc:1468] Processing CreateTablet for tablet cab4329d58404014833ae5029f8f296d (DEFAULT_TABLE table=default.Table_3947dfd6b9454b7eb988acee97d6e311 [id=e7367d6c3e05483cbff3ec85064df30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.743696 887 tablet_service.cc:1468] Processing CreateTablet for tablet 4c9efc7631df4ad29112a41f229ae2e9 (DEFAULT_TABLE table=default.Table_e6080511c8eb46c086fdb07159149641 [id=a268c90be9c3464fa38a63bef9c13547]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.744065 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab4329d58404014833ae5029f8f296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.744344 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9efc7631df4ad29112a41f229ae2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.746310 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 663588b7e9ee430faa180fec00897e22 (DEFAULT_TABLE table=default.Table_7e2035c1716744f7837845ba75f8b094 [id=4570d94b0b6f4e5db7ba8098dab81503]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.746750 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 0a90fd8cb29a4fffa65129dbca9ad84c (DEFAULT_TABLE table=default.Table_d568530b5b6e472fb3c3fd4b301a6dd2 [id=0f41315f3f4e46edb8f4b0f8a6b0f25a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.746750 896 tablet_service.cc:1468] Processing CreateTablet for tablet bd036ec08f774f5393067ce20bb07610 (DEFAULT_TABLE table=default.Table_4282c7f41da742dca77dc5965c226520 [id=2bfb3503e268489786dc3ff23eac100f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.747103 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd036ec08f774f5393067ce20bb07610. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.747128 1014 tablet_service.cc:1468] Processing CreateTablet for tablet cab4329d58404014833ae5029f8f296d (DEFAULT_TABLE table=default.Table_3947dfd6b9454b7eb988acee97d6e311 [id=e7367d6c3e05483cbff3ec85064df30b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.746482 895 tablet_service.cc:1468] Processing CreateTablet for tablet 3bc97a56aaf741648bae711cffacc6fb (DEFAULT_TABLE table=default.Table_675853c72fc2452eaad0d0a7dd62fd55 [id=ef87730d2fc94bf0860cddb9ee85b026]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.747551 755 tablet_service.cc:1468] Processing CreateTablet for tablet 189a9347cd5d4b5f955e5c0706aa3947 (DEFAULT_TABLE table=default.Table_a4182ebe832a4ef2a6b0a736bfcbc4b4 [id=b4a114b3b86a4c6ba3a8d3b69af100d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.747531 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 4c9efc7631df4ad29112a41f229ae2e9 (DEFAULT_TABLE table=default.Table_e6080511c8eb46c086fdb07159149641 [id=a268c90be9c3464fa38a63bef9c13547]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.747901 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189a9347cd5d4b5f955e5c0706aa3947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.747898 1021 tablet_service.cc:1468] Processing CreateTablet for tablet bd036ec08f774f5393067ce20bb07610 (DEFAULT_TABLE table=default.Table_4282c7f41da742dca77dc5965c226520 [id=2bfb3503e268489786dc3ff23eac100f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.748260 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3bc97a56aaf741648bae711cffacc6fb (DEFAULT_TABLE table=default.Table_675853c72fc2452eaad0d0a7dd62fd55 [id=ef87730d2fc94bf0860cddb9ee85b026]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.748620 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 6f5780d88dea41e7b855b41f12e96c52 (DEFAULT_TABLE table=default.Table_22678b3f1a3b4dd4afcafdc967bf2e93 [id=ddccd99845b2483a86638ca043ff9a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.748929 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 189a9347cd5d4b5f955e5c0706aa3947 (DEFAULT_TABLE table=default.Table_a4182ebe832a4ef2a6b0a736bfcbc4b4 [id=b4a114b3b86a4c6ba3a8d3b69af100d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.751603 747 tablet_service.cc:1468] Processing CreateTablet for tablet 4c9efc7631df4ad29112a41f229ae2e9 (DEFAULT_TABLE table=default.Table_e6080511c8eb46c086fdb07159149641 [id=a268c90be9c3464fa38a63bef9c13547]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.755482 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc97a56aaf741648bae711cffacc6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.756151 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 055f88e118454e32baa578cae9aceb0c
I20250625 13:51:58.757911 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 055f88e118454e32baa578cae9aceb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.758687 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 055f88e118454e32baa578cae9aceb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.758841 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d9462da7574a46839763b1c388102e60
I20250625 13:51:58.759559 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet d9462da7574a46839763b1c388102e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.759629 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3249c5c2aa564027a2c594ca4c31e1c8
I20250625 13:51:58.760293 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0b129117db064a15b27e82439637cf22
I20250625 13:51:58.760735 591 catalog_manager.cc:6652] ReplaceTablet: tablet 0b129117db064a15b27e82439637cf22 deleted and replaced by tablet 35aedb3b590d4917bac8fe0962d68fb3
I20250625 13:51:58.761768 593 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 (table default.Table_41e093f1c53c404fb4743c97d30f7f31 [id=13d38d05aad34e22884b9e07610b0e87]). Expected version 1 got 0
I20250625 13:51:58.761929 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.762171 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b129117db064a15b27e82439637cf22 with delete_type TABLET_DATA_DELETED (replaced by tablet 35aedb3b590d4917bac8fe0962d68fb3) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.762413 892 tablet_service.cc:1515] Processing DeleteTablet for tablet d9462da7574a46839763b1c388102e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.763188 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b129117db064a15b27e82439637cf22 with delete_type TABLET_DATA_DELETED (replaced by tablet 35aedb3b590d4917bac8fe0962d68fb3) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.763455 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.764154 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9efc7631df4ad29112a41f229ae2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.771096 753 tablet_service.cc:1468] Processing CreateTablet for tablet 6f5780d88dea41e7b855b41f12e96c52 (DEFAULT_TABLE table=default.Table_22678b3f1a3b4dd4afcafdc967bf2e93 [id=ddccd99845b2483a86638ca043ff9a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.771600 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 38a23b08a03b48eaa051f670eb7dac65
I20250625 13:51:58.772924 1512 raft_consensus.cc:491] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.773053 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.773100 1512 raft_consensus.cc:513] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.773608 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.773728 1512 leader_election.cc:290] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:58.774008 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a30a35e3639a43c5b078169060824723
I20250625 13:51:58.774365 596 catalog_manager.cc:6652] ReplaceTablet: tablet a30a35e3639a43c5b078169060824723 deleted and replaced by tablet efc9536a716f4b9f9fc4af1637b3cd80
I20250625 13:51:58.775266 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet a30a35e3639a43c5b078169060824723 with delete_type TABLET_DATA_DELETED (replaced by tablet efc9536a716f4b9f9fc4af1637b3cd80) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.775810 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 600d711975a74f66b9144758ad0dcfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.778540 896 tablet_service.cc:1515] Processing DeleteTablet for tablet a30a35e3639a43c5b078169060824723 with delete_type TABLET_DATA_DELETED (replaced by tablet efc9536a716f4b9f9fc4af1637b3cd80) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.779414 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef with delete_type TABLET_DATA_DELETED (replaced by tablet 88f72c555d9842f28ca4bf310da9c53c) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.786201 895 tablet_service.cc:1468] Processing CreateTablet for tablet ccce48011a15405db2b95c04c9b01b3c (DEFAULT_TABLE table=default.Table_817b97c6c8e742bf94f8c96a1485cf38 [id=f168fe273280497883707eb306f9c243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.786775 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccce48011a15405db2b95c04c9b01b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.786793 1028 tablet_service.cc:1468] Processing CreateTablet for tablet ccce48011a15405db2b95c04c9b01b3c (DEFAULT_TABLE table=default.Table_817b97c6c8e742bf94f8c96a1485cf38 [id=f168fe273280497883707eb306f9c243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.788851 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_08c36a3d2a03417982751548b18def03 [id=b8285b64bbee49f9be8741b252d03363] alter complete (version 2)
I20250625 13:51:58.789940 1012 tablet_service.cc:1468] Processing CreateTablet for tablet af704cdf44ef42f7b5ea10b199d0f017 (DEFAULT_TABLE table=default.Table_58618cdd2a7e4f508dbbb9a221a34800 [id=54dc596373954f3ca38f40af2e3d5d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.791059 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 48645c913fc24c03a922cf6ea7531022 (DEFAULT_TABLE table=default.Table_2e709eb4a94545c58de6814fff368a89 [id=8feb44e2430a4d92ac1296cd8c38ec72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.791733 879 tablet_service.cc:1468] Processing CreateTablet for tablet 6f5780d88dea41e7b855b41f12e96c52 (DEFAULT_TABLE table=default.Table_22678b3f1a3b4dd4afcafdc967bf2e93 [id=ddccd99845b2483a86638ca043ff9a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.792026 896 tablet_service.cc:1468] Processing CreateTablet for tablet 48645c913fc24c03a922cf6ea7531022 (DEFAULT_TABLE table=default.Table_2e709eb4a94545c58de6814fff368a89 [id=8feb44e2430a4d92ac1296cd8c38ec72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.792748 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5780d88dea41e7b855b41f12e96c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.792987 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48645c913fc24c03a922cf6ea7531022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.799032 1170 tablet_bootstrap.cc:492] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:58.799357 1170 ts_tablet_manager.cc:1397] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.094s user 0.009s sys 0.002s
I20250625 13:51:58.799480 752 tablet_service.cc:1468] Processing CreateTablet for tablet 3bc97a56aaf741648bae711cffacc6fb (DEFAULT_TABLE table=default.Table_675853c72fc2452eaad0d0a7dd62fd55 [id=ef87730d2fc94bf0860cddb9ee85b026]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.799916 756 tablet_service.cc:1468] Processing CreateTablet for tablet bd036ec08f774f5393067ce20bb07610 (DEFAULT_TABLE table=default.Table_4282c7f41da742dca77dc5965c226520 [id=2bfb3503e268489786dc3ff23eac100f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.800374 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5780d88dea41e7b855b41f12e96c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.800819 1170 raft_consensus.cc:357] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.801112 1170 raft_consensus.cc:383] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.801226 1170 raft_consensus.cc:738] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.801545 1170 consensus_queue.cc:260] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.802181 1170 ts_tablet_manager.cc:1428] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:51:58.803115 880 tablet_service.cc:1468] Processing CreateTablet for tablet af704cdf44ef42f7b5ea10b199d0f017 (DEFAULT_TABLE table=default.Table_58618cdd2a7e4f508dbbb9a221a34800 [id=54dc596373954f3ca38f40af2e3d5d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.804278 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc97a56aaf741648bae711cffacc6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.805051 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 055f88e118454e32baa578cae9aceb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.805526 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af704cdf44ef42f7b5ea10b199d0f017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.805783 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd036ec08f774f5393067ce20bb07610. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.809356 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_715c56d22d6c4746a9983d7f8670d23e" } new_table_name: "default.Table_93d4946c778542148fe4644b897607be" modify_external_catalogs: true
I20250625 13:51:58.812839 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_675c4ec332bd4560b7e40c2bc7ce5541" } new_table_name: "default.Table_1ffaf5826eda45159d7cef2f2505383a" modify_external_catalogs: true
I20250625 13:51:58.813365 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8603c0f87d5f40dab074f01aaf3d7c2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:58.813570 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_62ab081749614851902de10929497600" } new_table_name: "default.Table_b9f454bddd8a463b863eb0ca65ebf3af" modify_external_catalogs: true
W20250625 13:51:58.814026 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8603c0f87d5f40dab074f01aaf3d7c2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.814378 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9766d4ff7d994b58b92a788698344a8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.814931 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9766d4ff7d994b58b92a788698344a8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.815254 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_407b8d22e01548a4aadf917bd73fa8f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.815779 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_407b8d22e01548a4aadf917bd73fa8f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.816112 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_817b97c6c8e742bf94f8c96a1485cf38" } modify_external_catalogs: true
I20250625 13:51:58.816982 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_817b97c6c8e742bf94f8c96a1485cf38" } modify_external_catalogs: true
I20250625 13:51:58.821393 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "600d711975a74f66b9144758ad0dcfb0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:58.822652 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7641869a0bb463698244ff445d87a1f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.826556 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48645c913fc24c03a922cf6ea7531022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.828931 1167 ts_tablet_manager.cc:1918] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:58.829063 1167 log.cc:1199] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/1a7b72c2c464421eacfcf64008c3aea8
I20250625 13:51:58.829461 1167 ts_tablet_manager.cc:1939] T 1a7b72c2c464421eacfcf64008c3aea8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:58.829726 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af704cdf44ef42f7b5ea10b199d0f017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.829784 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 273) took 2499 ms. Trace:
I20250625 13:51:58.829887 1167 rpcz_store.cc:276] 0625 13:51:56.330746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.330792 (+ 46us) service_pool.cc:225] Handling call
0625 13:51:58.698168 (+2367376us) ts_tablet_manager.cc:1145] Deleting tablet 1a7b72c2c464421eacfcf64008c3aea8
0625 13:51:58.698176 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.796842 (+ 98666us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.828775 (+ 31933us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.829748 (+ 973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367144}
I20250625 13:51:58.830312 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 1a7b72c2c464421eacfcf64008c3aea8 (table default.Table_3599147b4b6b4aa3bcaa6de4475ae800 [id=cd3d67e4b72d4d919344482f7b10ed10]) successfully deleted
I20250625 13:51:58.831394 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 271) took 2502 ms. Trace:
I20250625 13:51:58.831526 1167 rpcz_store.cc:276] 0625 13:51:56.329085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.331521 (+ 2436us) service_pool.cc:225] Handling call
0625 13:51:58.831218 (+2499697us) ts_tablet_manager.cc:1145] Deleting tablet 36b151e200e1430c99ac194bc55881c1
0625 13:51:58.831226 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.831341 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2499445}
W20250625 13:51:58.831933 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 36b151e200e1430c99ac194bc55881c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36b151e200e1430c99ac194bc55881c1 already in progress: creating tablet
I20250625 13:51:58.832051 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 276) took 2180 ms. Trace:
I20250625 13:51:58.832180 1167 rpcz_store.cc:276] 0625 13:51:56.651478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.654030 (+ 2552us) service_pool.cc:225] Handling call
0625 13:51:58.831886 (+2177856us) ts_tablet_manager.cc:1145] Deleting tablet 01ba67b500b646d7aa55315f7ee23ed6
0625 13:51:58.831899 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.832014 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2170264}
W20250625 13:51:58.832496 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 01ba67b500b646d7aa55315f7ee23ed6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01ba67b500b646d7aa55315f7ee23ed6 already in progress: creating tablet
I20250625 13:51:58.832604 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 277) took 2180 ms. Trace:
I20250625 13:51:58.832710 1167 rpcz_store.cc:276] 0625 13:51:56.651698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.652238 (+ 540us) service_pool.cc:225] Handling call
0625 13:51:58.832488 (+2180250us) ts_tablet_manager.cc:1145] Deleting tablet 991bb428def545fd979e9f1a80ce4d18
0625 13:51:58.832496 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.832566 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153182}
W20250625 13:51:58.833055 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 991bb428def545fd979e9f1a80ce4d18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 991bb428def545fd979e9f1a80ce4d18 already in progress: creating tablet
I20250625 13:51:58.833173 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 287) took 2088 ms. Trace:
W20250625 13:51:58.833719 849 leader_election.cc:343] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.833740 1167 rpcz_store.cc:276] 0625 13:51:56.745029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.745137 (+ 108us) service_pool.cc:225] Handling call
0625 13:51:58.833031 (+2087894us) ts_tablet_manager.cc:1145] Deleting tablet bb1c7a85da844d34be805843bc60d350
0625 13:51:58.833039 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.833132 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2061265}
W20250625 13:51:58.834093 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bb1c7a85da844d34be805843bc60d350 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb1c7a85da844d34be805843bc60d350 already in progress: creating tablet
I20250625 13:51:58.813361 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4d8e666a80a345f7ba2ad05b84ce5a05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.836627 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d8e666a80a345f7ba2ad05b84ce5a05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.838495 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 286) took 2109 ms. Trace:
I20250625 13:51:58.838639 1167 rpcz_store.cc:276] 0625 13:51:56.729068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.729980 (+ 912us) service_pool.cc:225] Handling call
0625 13:51:58.838300 (+2108320us) ts_tablet_manager.cc:1145] Deleting tablet 3b0037d09796455cbaaadfaa9c010673
0625 13:51:58.838327 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.838447 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2046792}
W20250625 13:51:58.839112 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3b0037d09796455cbaaadfaa9c010673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0037d09796455cbaaadfaa9c010673 already in progress: creating tablet
I20250625 13:51:58.839172 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 288) took 2087 ms. Trace:
I20250625 13:51:58.839294 1167 rpcz_store.cc:276] 0625 13:51:56.752130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.752190 (+ 60us) service_pool.cc:225] Handling call
0625 13:51:58.839045 (+2086855us) ts_tablet_manager.cc:1145] Deleting tablet 643928ca4fc94ce58431a0525f2abb99
0625 13:51:58.839055 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.839131 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047101}
W20250625 13:51:58.839679 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 643928ca4fc94ce58431a0525f2abb99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 643928ca4fc94ce58431a0525f2abb99 already in progress: creating tablet
I20250625 13:51:58.814054 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8f116f4dda534af49710e06cfefb7439"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:58.840477 1167 tablet_replica.cc:331] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:58.840704 1167 raft_consensus.cc:2241] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:58.840853 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f116f4dda534af49710e06cfefb7439 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.840936 1167 raft_consensus.cc:2270] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:58.834177 849 leader_election.cc:343] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.846460 1171 tablet_bootstrap.cc:492] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.847489 1171 ts_tablet_manager.cc:1397] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.018s
I20250625 13:51:58.848562 1171 raft_consensus.cc:357] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.849792 1171 raft_consensus.cc:383] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.850055 1171 raft_consensus.cc:738] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.850517 1171 consensus_queue.cc:260] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.851351 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b129117db064a15b27e82439637cf22 with delete_type TABLET_DATA_DELETED (replaced by tablet 35aedb3b590d4917bac8fe0962d68fb3) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.861784 754 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:58.853405 851 leader_election.cc:343] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b7641869a0bb463698244ff445d87a1f
I20250625 13:51:58.862054 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.862226 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663588b7e9ee430faa180fec00897e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.862185 851 leader_election.cc:304] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:58.857357 896 tablet_service.cc:1515] Processing DeleteTablet for tablet b7641869a0bb463698244ff445d87a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.862793 1512 raft_consensus.cc:2747] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:58.863032 1171 ts_tablet_manager.cc:1428] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250625 13:51:58.863376 1171 tablet_bootstrap.cc:492] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.865271 1171 tablet_bootstrap.cc:654] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.865525 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a90fd8cb29a4fffa65129dbca9ad84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.869714 751 tablet_service.cc:1515] Processing DeleteTablet for tablet a30a35e3639a43c5b078169060824723 with delete_type TABLET_DATA_DELETED (replaced by tablet efc9536a716f4b9f9fc4af1637b3cd80) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.870908 743 tablet_service.cc:1468] Processing CreateTablet for tablet af704cdf44ef42f7b5ea10b199d0f017 (DEFAULT_TABLE table=default.Table_58618cdd2a7e4f508dbbb9a221a34800 [id=54dc596373954f3ca38f40af2e3d5d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.871048 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab4329d58404014833ae5029f8f296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.863411 740 tablet_service.cc:1515] Processing DeleteTablet for tablet d9462da7574a46839763b1c388102e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.872022 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 799f2edd960d45c1ace9195283a042e2 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c3f0ca6fa5482db7f57143911c838f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.872485 757 tablet_service.cc:1468] Processing CreateTablet for tablet ccce48011a15405db2b95c04c9b01b3c (DEFAULT_TABLE table=default.Table_817b97c6c8e742bf94f8c96a1485cf38 [id=f168fe273280497883707eb306f9c243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.873733 1171 tablet_bootstrap.cc:492] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:58.873884 1171 ts_tablet_manager.cc:1397] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250625 13:51:58.873979 748 tablet_service.cc:1468] Processing CreateTablet for tablet 48645c913fc24c03a922cf6ea7531022 (DEFAULT_TABLE table=default.Table_2e709eb4a94545c58de6814fff368a89 [id=8feb44e2430a4d92ac1296cd8c38ec72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:58.874336 1166 tablet_bootstrap.cc:492] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:58.874691 1171 raft_consensus.cc:357] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.874946 1171 raft_consensus.cc:383] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.875037 1171 raft_consensus.cc:738] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.875276 1171 consensus_queue.cc:260] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.875610 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9efc7631df4ad29112a41f229ae2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.875944 1171 ts_tablet_manager.cc:1428] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:58.876255 1171 tablet_bootstrap.cc:492] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:58.878494 1171 tablet_bootstrap.cc:654] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:58.879591 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd036ec08f774f5393067ce20bb07610. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.880491 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.880710 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.881667 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af704cdf44ef42f7b5ea10b199d0f017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.883234 1166 ts_tablet_manager.cc:1397] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.210s user 0.004s sys 0.002s
I20250625 13:51:58.883540 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc97a56aaf741648bae711cffacc6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.884102 1166 raft_consensus.cc:357] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.884292 1166 raft_consensus.cc:383] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:58.884354 1166 raft_consensus.cc:738] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:58.884588 1166 consensus_queue.cc:260] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.885162 1166 ts_tablet_manager.cc:1428] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:58.886912 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccce48011a15405db2b95c04c9b01b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.887924 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48645c913fc24c03a922cf6ea7531022. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.888331 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5780d88dea41e7b855b41f12e96c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.891523 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189a9347cd5d4b5f955e5c0706aa3947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.892639 1168 ts_tablet_manager.cc:1918] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:58.892858 1168 log.cc:1199] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a21335a06ec04da29b3351554c75f03c
I20250625 13:51:58.893450 1168 ts_tablet_manager.cc:1939] T a21335a06ec04da29b3351554c75f03c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:58.894934 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 168) took 4322 ms. Trace:
I20250625 13:51:58.895570 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccce48011a15405db2b95c04c9b01b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:58.897981 1166 tablet_bootstrap.cc:492] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:51:58.905696 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17b172c7631048ceb97ccbea896f7f5b has no consensus state
I20250625 13:51:58.906163 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9ce2182835d94395bb248b8d70c733d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:58.906327 591 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 4 got 0
W20250625 13:51:58.906769 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ce2182835d94395bb248b8d70c733d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.908478 1518 raft_consensus.cc:491] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.908596 1302 raft_consensus.cc:491] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.908726 1518 raft_consensus.cc:513] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:58.909305 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e77736191544e8da32ca9927f0a5804 consensus state has no leader
I20250625 13:51:58.909538 594 master_service.cc:948] ReplaceTablet: received request to replace tablet a22ade8cefac4140b08882c311ea5181 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:58.909580 1518 leader_election.cc:290] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.915870 1302 raft_consensus.cc:513] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.916602 1302 leader_election.cc:290] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.916749 1373 raft_consensus.cc:491] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.917739 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "799f2edd960d45c1ace9195283a042e2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.918054 1373 raft_consensus.cc:513] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.918681 1373 leader_election.cc:290] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:58.920122 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e281786e6e4a5d93b94b85b486655d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:58.920312 777 raft_consensus.cc:2466] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:51:58.927461 985 leader_election.cc:304] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:51:58.927806 1518 raft_consensus.cc:2802] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:58.927918 1518 raft_consensus.cc:491] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:58.928037 1518 raft_consensus.cc:3058] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.928630 1302 raft_consensus.cc:491] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.928769 1302 raft_consensus.cc:513] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:58.928838 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5ba76c130a4711af5b21e9eb8562e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:58.929356 711 leader_election.cc:343] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.930765 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_103b9969d6414619a95c5f6da350487c" } new_table_name: "default.Table_fb311c42df024421b325596e0cb7a778" modify_external_catalogs: true
I20250625 13:51:58.931047 1518 raft_consensus.cc:513] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.931630 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6820dad35d484763a041822f9b7bd57f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.932232 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6820dad35d484763a041822f9b7bd57f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.932981 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ccce48011a15405db2b95c04c9b01b3c
I20250625 13:51:58.933737 1518 leader_election.cc:290] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.933727 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f683f5d47e2c4c708cbe9cbbcca71ee6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.934235 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f683f5d47e2c4c708cbe9cbbcca71ee6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.935410 1302 leader_election.cc:290] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:51:58.936111 711 leader_election.cc:343] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.936750 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c72f9ae92a4537979fb479eb4467c4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.939440 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4dfad83f70604649b15f116a1d9aa9c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:58.939965 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dfad83f70604649b15f116a1d9aa9c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:58.941756 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_65410f67a07044e9a0b308de1bd53c34" } new_table_name: "default.Table_e90d64fdf05e442aa9da241d5fa1263b" modify_external_catalogs: true
I20250625 13:51:58.944749 1168 rpcz_store.cc:276] 0625 13:51:54.572218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.574920 (+ 2702us) service_pool.cc:225] Handling call
0625 13:51:58.631627 (+4056707us) ts_tablet_manager.cc:1145] Deleting tablet a21335a06ec04da29b3351554c75f03c
0625 13:51:58.631634 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.678354 (+ 46720us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.857109 (+178755us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.894883 (+ 37774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4056542}
I20250625 13:51:58.944604 1167 ts_tablet_manager.cc:1905] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.946190 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 583ee2f2c66841939c93104efc36facb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:58.946964 711 leader_election.cc:343] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.919018 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "799f2edd960d45c1ace9195283a042e2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:58.950111 1168 tablet_replica.cc:331] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:58.919356 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5ba76c130a4711af5b21e9eb8562e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:58.950275 914 raft_consensus.cc:2466] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:58.950340 1168 raft_consensus.cc:2241] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.925637 1512 raft_consensus.cc:491] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.951030 1512 raft_consensus.cc:513] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.951704 1512 leader_election.cc:290] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:58.933461 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e281786e6e4a5d93b94b85b486655d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:58.934057 896 tablet_service.cc:1515] Processing DeleteTablet for tablet ccce48011a15405db2b95c04c9b01b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:58.952019 1545 tablet.cc:1722] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:51:58.937716 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c72f9ae92a4537979fb479eb4467c4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:58.952109 915 raft_consensus.cc:3058] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:58.910475 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e281786e6e4a5d93b94b85b486655d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:58.948107 1510 consensus_queue.cc:1035] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:58.954519 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "389d81329b8d430a9098b0630612fdcb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:58.955070 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet a21335a06ec04da29b3351554c75f03c (table default.Table_1329da083f314d9d8fa3ac2e851646bf [id=3286dc94d1ba4645beb4a2a54bec8c8a]) successfully deleted
I20250625 13:51:58.956008 915 raft_consensus.cc:2466] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:51:58.956409 849 leader_election.cc:343] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.956467 983 leader_election.cc:304] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:58.952114 1168 raft_consensus.cc:2270] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.957386 1165 ts_tablet_manager.cc:1918] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:58.957561 1165 log.cc:1199] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/87327f74beb845768bbfd5ae6d2bcf88
I20250625 13:51:58.957620 1518 raft_consensus.cc:2802] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:58.958055 1165 ts_tablet_manager.cc:1939] T 87327f74beb845768bbfd5ae6d2bcf88 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:51:58.958344 711 leader_election.cc:343] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.958475 711 leader_election.cc:304] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:58.958925 711 leader_election.cc:304] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:58.959053 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e281786e6e4a5d93b94b85b486655d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:58.959251 777 raft_consensus.cc:3058] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:58.959383 711 leader_election.cc:343] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:58.961457 711 leader_election.cc:304] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:58.961505 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c54f67fabc114fc6bb555987a103adba" } modify_external_catalogs: true
I20250625 13:51:58.961709 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c54f67fabc114fc6bb555987a103adba" } modify_external_catalogs: true
I20250625 13:51:58.962224 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cf29f4b4bbb94091affd4440a4b8447c" } modify_external_catalogs: true
I20250625 13:51:58.962366 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cf29f4b4bbb94091affd4440a4b8447c" } modify_external_catalogs: true
I20250625 13:51:58.967847 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ccce48011a15405db2b95c04c9b01b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:58.969277 1302 raft_consensus.cc:2747] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:58.973215 1302 raft_consensus.cc:2747] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:58.973910 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 302) took 2263 ms. Trace:
I20250625 13:51:58.974678 1165 rpcz_store.cc:276] 0625 13:51:56.710313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.710364 (+ 51us) service_pool.cc:225] Handling call
0625 13:51:58.612970 (+1902606us) ts_tablet_manager.cc:1145] Deleting tablet 87327f74beb845768bbfd5ae6d2bcf88
0625 13:51:58.612979 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:58.703058 (+ 90079us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.928475 (+225417us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:58.973864 (+ 45389us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1902312}
I20250625 13:51:58.975553 1302 raft_consensus.cc:491] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:58.976366 1302 raft_consensus.cc:513] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:58.977039 1302 leader_election.cc:290] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:58.978861 1165 tablet_replica.cc:331] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:58.979465 1165 raft_consensus.cc:2241] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:58.980276 1165 raft_consensus.cc:2270] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:58.980641 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c1a98dc40684954a602567cf3476879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.982841 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_08c36a3d2a03417982751548b18def03" } modify_external_catalogs: true
I20250625 13:51:58.983059 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_08c36a3d2a03417982751548b18def03" } modify_external_catalogs: true
I20250625 13:51:58.985512 1165 ts_tablet_manager.cc:1905] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:58.981168 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet cbaa018753d8455199f3869323fb5582 with delete_type TABLET_DATA_DELETED (replaced by tablet cd4f0857f52449e1b26fde09c6c7c9d4) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:58.956967 851 leader_election.cc:343] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 389d81329b8d430a9098b0630612fdcb
I20250625 13:51:58.990247 851 leader_election.cc:304] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:58.990597 1442 raft_consensus.cc:2747] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:58.991775 849 consensus_peers.cc:487] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:58.993628 1168 ts_tablet_manager.cc:1905] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:59.004153 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be3313ec5dfb41b68662d3ff4c8c94f9 consensus state has no leader
I20250625 13:51:59.004817 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_21f2f615829242bfb6460e42b5995d14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.005434 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21f2f615829242bfb6460e42b5995d14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.008189 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d26820eaba98401da3a8b5348be21fc6 consensus state has no leader
I20250625 13:51:59.008518 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0d3b87078b494559b4a686a10a478402" } modify_external_catalogs: true
I20250625 13:51:59.008728 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0d3b87078b494559b4a686a10a478402" } modify_external_catalogs: true
I20250625 13:51:59.015273 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d4f19ddc51324cc18242c6081f3f7de5
I20250625 13:51:59.022687 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d4f19ddc51324cc18242c6081f3f7de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.023962 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet ccce48011a15405db2b95c04c9b01b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:58.993074 1564 tablet.cc:1722] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:51:59.007366 1373 raft_consensus.cc:2802] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:59.024602 1373 raft_consensus.cc:491] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:59.011195 777 raft_consensus.cc:2466] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:51:59.012179 1166 tablet_bootstrap.cc:654] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.025614 1373 raft_consensus.cc:3058] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.027716 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 9ba63184ca2b4badafcf08309ef75164 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:59.029292 1170 tablet_bootstrap.cc:492] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:59.032042 1606 raft_consensus.cc:491] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.032598 1606 raft_consensus.cc:513] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.034226 1606 leader_election.cc:290] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:59.037149 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbaa018753d8455199f3869323fb5582" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:59.037426 915 raft_consensus.cc:2391] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:51:59.039389 1171 tablet_bootstrap.cc:492] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:59.039564 1171 ts_tablet_manager.cc:1397] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.163s user 0.035s sys 0.074s
I20250625 13:51:59.040467 1171 raft_consensus.cc:357] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.040685 1171 raft_consensus.cc:383] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.040751 1171 raft_consensus.cc:738] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.040988 1171 consensus_queue.cc:260] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.041939 1171 ts_tablet_manager.cc:1428] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:59.042272 1171 tablet_bootstrap.cc:492] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:59.042142 1373 raft_consensus.cc:513] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.038054 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet d4f19ddc51324cc18242c6081f3f7de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.038272 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e281786e6e4a5d93b94b85b486655d with delete_type TABLET_DATA_DELETED (replaced by tablet be3313ec5dfb41b68662d3ff4c8c94f9) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.042676 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
W20250625 13:51:59.042070 985 leader_election.cc:343] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cbaa018753d8455199f3869323fb5582
I20250625 13:51:59.042989 985 leader_election.cc:304] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.043246 1606 raft_consensus.cc:3058] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.041272 1170 tablet_bootstrap.cc:654] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.045135 1373 leader_election.cc:290] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:59.046036 753 tablet_service.cc:1515] Processing DeleteTablet for tablet d4f19ddc51324cc18242c6081f3f7de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.047492 1606 raft_consensus.cc:2747] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:59.053421 1171 tablet_bootstrap.cc:654] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.053395 593 master_service.cc:948] ReplaceTablet: received request to replace tablet d26820eaba98401da3a8b5348be21fc6 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:59.053839 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd] alter complete (version 4)
I20250625 13:51:59.054400 1518 raft_consensus.cc:695] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:51:59.054713 1518 consensus_queue.cc:237] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.063886 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 583ee2f2c66841939c93104efc36facb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.061195 1170 tablet_bootstrap.cc:492] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:59.064325 1170 ts_tablet_manager.cc:1397] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
W20250625 13:51:59.072077 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 112690us
I20250625 13:51:59.072955 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9bb7ebc40a14ecaa0c692a8e1117db3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:59.073598 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9bb7ebc40a14ecaa0c692a8e1117db3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:59.073796 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5ba76c130a4711af5b21e9eb8562e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:59.073998 913 raft_consensus.cc:3058] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.076895 913 raft_consensus.cc:2466] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:59.078148 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5ba76c130a4711af5b21e9eb8562e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:59.079170 1170 raft_consensus.cc:357] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.079769 1166 tablet_bootstrap.cc:492] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.079934 1166 ts_tablet_manager.cc:1397] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.194s user 0.002s sys 0.003s
I20250625 13:51:59.080250 1170 raft_consensus.cc:383] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.080829 1166 raft_consensus.cc:357] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.081022 1166 raft_consensus.cc:383] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.081130 1166 raft_consensus.cc:738] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.081166 1170 raft_consensus.cc:738] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.081499 1170 consensus_queue.cc:260] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.081476 1166 consensus_queue.cc:260] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:59.081993 711 leader_election.cc:343] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.082486 1170 ts_tablet_manager.cc:1428] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 13:51:59.084211 1166 ts_tablet_manager.cc:1428] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:59.082769 711 leader_election.cc:304] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:59.085495 1373 raft_consensus.cc:2802] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:59.085804 1373 raft_consensus.cc:695] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:59.085948 1166 tablet_bootstrap.cc:492] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:59.086314 1373 consensus_queue.cc:237] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:59.088124 711 leader_election.cc:343] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.088332 711 leader_election.cc:304] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.088855 1373 raft_consensus.cc:2747] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:59.089063 711 leader_election.cc:343] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.090363 1166 tablet_bootstrap.cc:654] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.093935 1170 tablet_bootstrap.cc:492] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:59.097242 1170 tablet_bootstrap.cc:654] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.102592 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 34ab663b7eba452c9f9775c0f568010b from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:59.103207 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 600d711975a74f66b9144758ad0dcfb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.104171 1171 tablet_bootstrap.cc:492] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:59.104880 1171 ts_tablet_manager.cc:1397] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.001s
I20250625 13:51:59.109351 1512 raft_consensus.cc:491] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.109586 1512 raft_consensus.cc:513] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.110419 1512 leader_election.cc:290] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:59.111490 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02f2a2b595bc43309a29b3f1bc5d3766" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:59.111788 776 raft_consensus.cc:2466] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:59.111864 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02f2a2b595bc43309a29b3f1bc5d3766" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:59.112607 849 leader_election.cc:343] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.113210 851 leader_election.cc:304] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:59.113806 1512 raft_consensus.cc:2802] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:59.113942 1512 raft_consensus.cc:491] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:59.114044 1512 raft_consensus.cc:3058] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.117089 1165 ts_tablet_manager.cc:1918] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:51:59.117257 1165 log.cc:1199] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/412a0c5f05ab489ab9f8fc3f44fa7bdc
I20250625 13:51:59.117689 1165 ts_tablet_manager.cc:1939] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.118031 1512 raft_consensus.cc:513] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.118643 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 303) took 2385 ms. Trace:
I20250625 13:51:59.118762 1165 rpcz_store.cc:276] 0625 13:51:56.732772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.732955 (+ 183us) service_pool.cc:225] Handling call
0625 13:51:58.978771 (+2245816us) ts_tablet_manager.cc:1145] Deleting tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc
0625 13:51:58.978786 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.053312 (+ 74526us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.116939 (+ 63627us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.118576 (+ 1637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245595}
I20250625 13:51:59.119405 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02f2a2b595bc43309a29b3f1bc5d3766" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:59.119642 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]) successfully deleted
W20250625 13:51:59.120172 849 leader_election.cc:343] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.121308 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02f2a2b595bc43309a29b3f1bc5d3766" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:59.121631 776 raft_consensus.cc:3058] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.122933 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 34ab663b7eba452c9f9775c0f568010b
I20250625 13:51:59.123510 591 catalog_manager.cc:6652] ReplaceTablet: tablet 34ab663b7eba452c9f9775c0f568010b deleted and replaced by tablet 155737a8d5604eb69524c4d6bea61550
I20250625 13:51:59.123483 1168 ts_tablet_manager.cc:1918] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 13:51:59.123638 1168 log.cc:1199] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/76f074156559406c875e65796e6b5ee6
I20250625 13:51:59.124775 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ab663b7eba452c9f9775c0f568010b with delete_type TABLET_DATA_DELETED (replaced by tablet 155737a8d5604eb69524c4d6bea61550) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.125696 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ab663b7eba452c9f9775c0f568010b with delete_type TABLET_DATA_DELETED (replaced by tablet 155737a8d5604eb69524c4d6bea61550) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.125900 776 raft_consensus.cc:2466] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:59.126644 1167 ts_tablet_manager.cc:1918] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:51:59.126798 1167 log.cc:1199] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/412a0c5f05ab489ab9f8fc3f44fa7bdc
I20250625 13:51:59.131821 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e0ded9bf53314d66a9363d735e000f5c" } new_table_name: "default.Table_340984100eaf438ab7fb71232d7fe268" modify_external_catalogs: true
I20250625 13:51:59.131877 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_58618cdd2a7e4f508dbbb9a221a34800" } new_table_name: "default.Table_db99293d790941fcbd2bbb8a9ed7967c" modify_external_catalogs: true
I20250625 13:51:59.132143 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8319fc497f694699988a5dfd8de276e0" } new_table_name: "default.Table_f02770cf5ed24271b290c8224d3675d6" modify_external_catalogs: true
I20250625 13:51:59.136834 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3859f399a7b84de99658380bd6c24ff5
I20250625 13:51:59.139021 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3859f399a7b84de99658380bd6c24ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.138208 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 3859f399a7b84de99658380bd6c24ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.127287 1167 ts_tablet_manager.cc:1939] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:59.140769 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_075afd49a8144b2eab9d29164f19a5a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.141333 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_075afd49a8144b2eab9d29164f19a5a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.142710 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 296) took 2106 ms. Trace:
I20250625 13:51:59.142897 1167 rpcz_store.cc:276] 0625 13:51:57.035913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.036446 (+ 533us) service_pool.cc:225] Handling call
0625 13:51:58.840407 (+1803961us) ts_tablet_manager.cc:1145] Deleting tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc
0625 13:51:58.840415 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.059666 (+219251us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.126521 (+ 66855us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.142655 (+ 16134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795752}
I20250625 13:51:59.143108 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_279ed5effac34385a5eb2ea2439ec1ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.143664 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_279ed5effac34385a5eb2ea2439ec1ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.143672 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]) successfully deleted
I20250625 13:51:59.145653 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_89e73d4233d14f7ab1805e3351155943" } modify_external_catalogs: true
I20250625 13:51:59.145845 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_89e73d4233d14f7ab1805e3351155943" } modify_external_catalogs: true
I20250625 13:51:59.145996 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 297) took 2108 ms. Trace:
I20250625 13:51:59.119257 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 304) took 2382 ms. Trace:
I20250625 13:51:59.146267 1165 rpcz_store.cc:276] 0625 13:51:56.736906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.738127 (+ 1221us) service_pool.cc:225] Handling call
0625 13:51:59.119069 (+2380942us) ts_tablet_manager.cc:1145] Deleting tablet 6c5fc63828d9408b9cf774a966f43cf2
0625 13:51:59.119085 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.119197 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2380465}
I20250625 13:51:59.146735 1165 tablet_replica.cc:331] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:59.124979 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ab663b7eba452c9f9775c0f568010b with delete_type TABLET_DATA_DELETED (replaced by tablet 155737a8d5604eb69524c4d6bea61550) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.125669 1512 leader_election.cc:290] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:59.126482 851 leader_election.cc:304] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:59.147354 1168 ts_tablet_manager.cc:1939] T 76f074156559406c875e65796e6b5ee6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:59.147630 1165 raft_consensus.cc:2241] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.147903 1165 raft_consensus.cc:2270] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.147706 1510 raft_consensus.cc:2802] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:59.148125 1510 raft_consensus.cc:695] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:59.137822 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 3859f399a7b84de99658380bd6c24ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:51:59.148243 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6c5fc63828d9408b9cf774a966f43cf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c5fc63828d9408b9cf774a966f43cf2 already in progress: creating tablet
I20250625 13:51:59.148452 1510 consensus_queue.cc:237] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:59.150727 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dad4ea8cf44944b08f53428c95809daf consensus state has no leader
W20250625 13:51:59.150877 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af704cdf44ef42f7b5ea10b199d0f017 consensus state has no leader
I20250625 13:51:59.151170 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1452b75693a442568546fb6e23ff895e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.151337 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_722e0f5afe094da88dd5c6985558f1c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.151746 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1452b75693a442568546fb6e23ff895e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.151930 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_722e0f5afe094da88dd5c6985558f1c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.155143 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 179) took 4455 ms. Trace:
I20250625 13:51:59.156009 1165 ts_tablet_manager.cc:1905] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:59.156518 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 430c3bed360b4f32855a8adb99076608 consensus state has no leader
I20250625 13:51:59.156518 1168 rpcz_store.cc:276] 0625 13:51:54.699601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.700788 (+ 1187us) service_pool.cc:225] Handling call
0625 13:51:58.950027 (+4249239us) ts_tablet_manager.cc:1145] Deleting tablet 76f074156559406c875e65796e6b5ee6
0625 13:51:58.950039 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.119649 (+169610us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.123368 (+ 3719us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.155087 (+ 31719us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4248983}
I20250625 13:51:59.156935 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ee7ffdafe40d48bc89ffa9a18595cf9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.157644 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee7ffdafe40d48bc89ffa9a18595cf9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.157670 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 76f074156559406c875e65796e6b5ee6 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]) successfully deleted
I20250625 13:51:59.159130 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85834f366fa42c294c9290af21ea111. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:59.159399 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:51:59.159716 1608 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d85834f366fa42c294c9290af21ea111)
I20250625 13:51:59.160043 1608 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:51:59.163614 1168 tablet_replica.cc:331] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:59.164180 1168 raft_consensus.cc:2241] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.166323 1168 raft_consensus.cc:2270] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.174144 597 master_service.cc:948] ReplaceTablet: received request to replace tablet ac4d368347444846880954ced9651404 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:59.175988 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2de9656958164dc4ae2fb0078b38f6c3
I20250625 13:51:59.176620 1608 leader_election.cc:290] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58351267767f4d9a85f974ebc40db6fe (127.0.104.61:45727), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
I20250625 13:51:59.177069 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.177474 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.183427 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 50786a4c25274ffb8f9910c5dafe3933 (table default.Table_55953048150049c490cd5b9346184cd0 [id=d7cbe338b2554a6e99b07325632aaa33]). Expected version 2 got 0
I20250625 13:51:59.186197 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_984c484e0c964d3889777159c75343a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.186508 1473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46832 (request call id 0) took 1318 ms. Trace:
W20250625 13:51:59.186842 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_984c484e0c964d3889777159c75343a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.190028 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.190286 1167 rpcz_store.cc:276] 0625 13:51:57.036054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.036474 (+ 420us) service_pool.cc:225] Handling call
0625 13:51:59.144802 (+2108328us) ts_tablet_manager.cc:1145] Deleting tablet 6c5fc63828d9408b9cf774a966f43cf2
0625 13:51:59.144813 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.144948 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2099991}
I20250625 13:51:59.183511 1168 ts_tablet_manager.cc:1905] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.197007 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 69c44d72516e4883aff636ca65929c11
I20250625 13:51:59.199154 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 69c44d72516e4883aff636ca65929c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.199855 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 69c44d72516e4883aff636ca65929c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.192502 1518 consensus_queue.cc:1035] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20250625 13:51:59.198467 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 69c44d72516e4883aff636ca65929c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:51:59.201655 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6c5fc63828d9408b9cf774a966f43cf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c5fc63828d9408b9cf774a966f43cf2 already in progress: creating tablet
I20250625 13:51:59.202111 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 299) took 2165 ms. Trace:
I20250625 13:51:59.205710 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_af0069799827448399dea82497b0b4d3" } new_table_name: "default.Table_2ed47131082c4d11b7ba70c6432bc36c" modify_external_catalogs: true
I20250625 13:51:59.209899 1166 tablet_bootstrap.cc:492] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.210141 1166 ts_tablet_manager.cc:1397] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.124s user 0.031s sys 0.035s
I20250625 13:51:59.210959 1166 raft_consensus.cc:357] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.211344 1166 raft_consensus.cc:383] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.211489 1166 raft_consensus.cc:738] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.211851 1166 consensus_queue.cc:260] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.214354 1166 ts_tablet_manager.cc:1428] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:59.211983 748 tablet_service.cc:1468] Processing CreateTablet for tablet 45d68de75ac84f1a8138c38fd51547cf (DEFAULT_TABLE table=default.Table_9150143ab7394f1baf2fb087e8265dfa [id=f60ff5e9620e422d9bb3b03ec34e1f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.214742 1166 tablet_bootstrap.cc:492] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:59.219758 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.215090 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d68de75ac84f1a8138c38fd51547cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.202593 983 consensus_peers.cc:487] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:59.209812 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 45d68de75ac84f1a8138c38fd51547cf (DEFAULT_TABLE table=default.Table_9150143ab7394f1baf2fb087e8265dfa [id=f60ff5e9620e422d9bb3b03ec34e1f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.221014 1167 rpcz_store.cc:276] 0625 13:51:57.036273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.036610 (+ 337us) service_pool.cc:225] Handling call
0625 13:51:59.201932 (+2165322us) ts_tablet_manager.cc:1145] Deleting tablet df1ed922750f45eab02d1fc0c26af75e
0625 13:51:59.201943 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.202053 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2156051}
I20250625 13:51:59.221393 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d68de75ac84f1a8138c38fd51547cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.221509 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet df1ed922750f45eab02d1fc0c26af75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df1ed922750f45eab02d1fc0c26af75e already in progress: creating tablet
I20250625 13:51:59.221951 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 298) took 2185 ms. Trace:
I20250625 13:51:59.222048 1167 rpcz_store.cc:276] 0625 13:51:57.036162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.036581 (+ 419us) service_pool.cc:225] Handling call
0625 13:51:59.221784 (+2185203us) ts_tablet_manager.cc:1145] Deleting tablet 46316eb725bc47e695af476c0b185504
0625 13:51:59.221794 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.221889 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2175458}
W20250625 13:51:59.222471 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 46316eb725bc47e695af476c0b185504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46316eb725bc47e695af476c0b185504 already in progress: creating tablet
I20250625 13:51:59.222649 1166 tablet_bootstrap.cc:654] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.222903 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 305) took 2165 ms. Trace:
I20250625 13:51:59.223035 1167 rpcz_store.cc:276] 0625 13:51:57.057515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.057589 (+ 74us) service_pool.cc:225] Handling call
0625 13:51:59.222702 (+2165113us) ts_tablet_manager.cc:1145] Deleting tablet 11711546f0e14f9a9eabf003f49a69b7
0625 13:51:59.222713 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.222827 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2164915}
W20250625 13:51:59.223484 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 11711546f0e14f9a9eabf003f49a69b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11711546f0e14f9a9eabf003f49a69b7 already in progress: creating tablet
I20250625 13:51:59.220130 1047 raft_consensus.cc:3058] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.223845 1167 tablet_replica.cc:331] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:59.224035 1167 raft_consensus.cc:2241] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.224205 1167 raft_consensus.cc:2270] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.224748 1167 ts_tablet_manager.cc:1905] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.225368 1171 raft_consensus.cc:357] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.225590 1171 raft_consensus.cc:383] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.225669 1171 raft_consensus.cc:738] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.225925 1171 consensus_queue.cc:260] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.226435 1171 ts_tablet_manager.cc:1428] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.121s user 0.001s sys 0.001s
I20250625 13:51:59.226776 1171 tablet_bootstrap.cc:492] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:59.232224 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8f116f4dda534af49710e06cfefb7439" } new_table_name: "default.Table_da32ce790d1e43469d2ece95ae454446" modify_external_catalogs: true
I20250625 13:51:59.233695 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cc7e074a01254cbb9272fd4ea7dba124" } modify_external_catalogs: true
I20250625 13:51:59.233872 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cc7e074a01254cbb9272fd4ea7dba124" } modify_external_catalogs: true
I20250625 13:51:59.234822 1166 tablet_bootstrap.cc:492] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.235018 1166 ts_tablet_manager.cc:1397] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20250625 13:51:59.235419 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cc3ae4d4f62d47af9a510dab1dd32ff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.236016 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc3ae4d4f62d47af9a510dab1dd32ff5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.236765 1166 raft_consensus.cc:357] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.237000 1166 raft_consensus.cc:383] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.237159 1166 raft_consensus.cc:738] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.237243 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0965fa983950458882df05ba9d414275"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.237519 1166 consensus_queue.cc:260] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:59.237758 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0965fa983950458882df05ba9d414275 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.238831 1166 ts_tablet_manager.cc:1428] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 13:51:59.239216 1166 tablet_bootstrap.cc:492] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:59.240720 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1493c363105b457a9b8b6a7673039e85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.240933 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 446 } } ignore_live_leader: false dest_uuid: "58351267767f4d9a85f974ebc40db6fe" is_pre_election: true
W20250625 13:51:59.241261 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1493c363105b457a9b8b6a7673039e85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.243762 1166 tablet_bootstrap.cc:654] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.246102 1620 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250625 13:51:59.246816 1165 ts_tablet_manager.cc:1918] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.247243 1165 log.cc:1199] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/8149aea0ff7646d3b94c9159f56e48ef
I20250625 13:51:59.247757 1165 ts_tablet_manager.cc:1939] T 8149aea0ff7646d3b94c9159f56e48ef P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.248028 1455 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" instance_seqno: 1750859510146049) as {username='slave'} at 127.0.104.1:45127; Asking this server to re-register.
I20250625 13:51:59.248664 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 305) took 2505 ms. Trace:
I20250625 13:51:59.249277 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e54b0d212c3542cc8cb5fb2e997f6be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.249253 823 heartbeater.cc:461] Registering TS with master...
W20250625 13:51:59.250003 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e54b0d212c3542cc8cb5fb2e997f6be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.249346 1165 rpcz_store.cc:276] 0625 13:51:56.743148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.743509 (+ 361us) service_pool.cc:225] Handling call
0625 13:51:59.146643 (+2403134us) ts_tablet_manager.cc:1145] Deleting tablet 8149aea0ff7646d3b94c9159f56e48ef
0625 13:51:59.146656 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.161175 (+ 14519us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.246667 (+ 85492us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.248620 (+ 1953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2402832}
I20250625 13:51:59.250989 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 306) took 2482 ms. Trace:
I20250625 13:51:59.251260 1165 rpcz_store.cc:276] 0625 13:51:56.768978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.769035 (+ 57us) service_pool.cc:225] Handling call
0625 13:51:59.250830 (+2481795us) ts_tablet_manager.cc:1145] Deleting tablet 46316eb725bc47e695af476c0b185504
0625 13:51:59.250839 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.250945 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2481563}
I20250625 13:51:59.251817 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.251896 1455 ts_manager.cc:194] Registered new tserver with Master: 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:59.251925 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 8149aea0ff7646d3b94c9159f56e48ef (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]) successfully deleted
W20250625 13:51:59.252362 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 46316eb725bc47e695af476c0b185504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46316eb725bc47e695af476c0b185504 already in progress: creating tablet
I20250625 13:51:59.252390 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 307) took 2469 ms. Trace:
I20250625 13:51:59.252542 1165 rpcz_store.cc:276] 0625 13:51:56.782770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.782845 (+ 75us) service_pool.cc:225] Handling call
0625 13:51:59.252257 (+2469412us) ts_tablet_manager.cc:1145] Deleting tablet df1ed922750f45eab02d1fc0c26af75e
0625 13:51:59.252266 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.252360 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2469160}
I20250625 13:51:59.252894 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 446 } } ignore_live_leader: false dest_uuid: "d85834f366fa42c294c9290af21ea111" is_pre_election: true
I20250625 13:51:59.253127 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 309) took 2464 ms. Trace:
I20250625 13:51:59.253262 1165 rpcz_store.cc:276] 0625 13:51:56.788370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.788418 (+ 48us) service_pool.cc:225] Handling call
0625 13:51:59.252954 (+2464536us) ts_tablet_manager.cc:1145] Deleting tablet 7ba1338d0562418eadabdb8dbcaedee2
0625 13:51:59.252962 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.253098 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2464298}
W20250625 13:51:59.254140 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet df1ed922750f45eab02d1fc0c26af75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df1ed922750f45eab02d1fc0c26af75e already in progress: creating tablet
I20250625 13:51:59.254561 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 308) took 2466 ms. Trace:
I20250625 13:51:59.254984 1165 rpcz_store.cc:276] 0625 13:51:56.788226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.788697 (+ 471us) service_pool.cc:225] Handling call
0625 13:51:59.254448 (+2465751us) ts_tablet_manager.cc:1145] Deleting tablet 29be8c0350584de382985e66556bdcca
0625 13:51:59.254457 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.254530 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2465548}
W20250625 13:51:59.255299 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7ba1338d0562418eadabdb8dbcaedee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba1338d0562418eadabdb8dbcaedee2 already in progress: creating tablet
I20250625 13:51:59.255693 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d26820eaba98401da3a8b5348be21fc6
W20250625 13:51:59.255913 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 29be8c0350584de382985e66556bdcca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29be8c0350584de382985e66556bdcca already in progress: creating tablet
I20250625 13:51:59.256177 593 catalog_manager.cc:6652] ReplaceTablet: tablet d26820eaba98401da3a8b5348be21fc6 deleted and replaced by tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57
I20250625 13:51:59.253729 1421 leader_election.cc:304] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fec9d8a79054a5991ae90d8a3101e32; no voters: 58351267767f4d9a85f974ebc40db6fe, d85834f366fa42c294c9290af21ea111
I20250625 13:51:59.256871 748 tablet_service.cc:1515] Processing DeleteTablet for tablet d26820eaba98401da3a8b5348be21fc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.257025 1608 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:51:59.258538 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 312) took 2416 ms. Trace:
I20250625 13:51:59.258989 1165 rpcz_store.cc:276] 0625 13:51:56.842195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.842249 (+ 54us) service_pool.cc:225] Handling call
0625 13:51:59.255503 (+2413254us) ts_tablet_manager.cc:1145] Deleting tablet 11711546f0e14f9a9eabf003f49a69b7
0625 13:51:59.258408 (+ 2905us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.258504 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413000}
I20250625 13:51:59.259430 1165 tablet_replica.cc:331] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:59.260399 1165 raft_consensus.cc:2241] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.260954 1165 raft_consensus.cc:2270] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.261590 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0012e95ff4924723a6e6aaf6b85da438 (DEFAULT_TABLE table=default.Table_af0069799827448399dea82497b0b4d3 [id=c0d109f727f14a0eb8abd4ca7ff94dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.261916 1165 ts_tablet_manager.cc:1905] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.262068 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0012e95ff4924723a6e6aaf6b85da438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.267966 743 tablet_service.cc:1468] Processing CreateTablet for tablet 1310fee26fe642bf9919a2bd8e1c091f (DEFAULT_TABLE table=default.Table_f4aa1ee5e69c4dd6a8917a9a9900768f [id=ad66b451137740d1be88d600e320e218]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.268334 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1310fee26fe642bf9919a2bd8e1c091f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.271930 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 430c3bed360b4f32855a8adb99076608 from {username='slave'} at 127.0.0.1:54762
W20250625 13:51:59.275425 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 11711546f0e14f9a9eabf003f49a69b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11711546f0e14f9a9eabf003f49a69b7 already in progress: creating tablet
I20250625 13:51:59.276571 1165 ts_tablet_manager.cc:1918] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:51:59.278201 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0012e95ff4924723a6e6aaf6b85da438 consensus state has no leader
I20250625 13:51:59.278661 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet fd39216b01054ff38ec9175d7a2f87f2
W20250625 13:51:59.279534 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4be0bc23cde4f0f94bd678096e0c7d8 has no consensus state
I20250625 13:51:59.279764 1166 tablet_bootstrap.cc:492] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.280052 1166 ts_tablet_manager.cc:1397] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.003s
I20250625 13:51:59.281073 1166 raft_consensus.cc:357] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.281373 1166 raft_consensus.cc:383] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.281498 1166 raft_consensus.cc:738] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.281845 1166 consensus_queue.cc:260] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.283874 1166 ts_tablet_manager.cc:1428] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:59.284235 1166 tablet_bootstrap.cc:492] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:59.291203 1166 tablet_bootstrap.cc:654] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.291410 1165 log.cc:1199] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d9bb7ebc40a14ecaa0c692a8e1117db3
I20250625 13:51:59.292013 743 tablet_service.cc:1468] Processing CreateTablet for tablet 2c1e362b69fe40c0b6472b56f8cb9bcb (DEFAULT_TABLE table=default.Table_0376b3d69a954c6881b4fd9d11e4b86b [id=0d141ebb7fc74195a5c05eeb73708142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.292595 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1e362b69fe40c0b6472b56f8cb9bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.297503 748 tablet_service.cc:1515] Processing DeleteTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.298338 1165 ts_tablet_manager.cc:1939] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.301481 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 328) took 2326 ms. Trace:
I20250625 13:51:59.301635 1165 rpcz_store.cc:276] 0625 13:51:56.975466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.975659 (+ 193us) service_pool.cc:225] Handling call
0625 13:51:59.259361 (+2283702us) ts_tablet_manager.cc:1145] Deleting tablet d9bb7ebc40a14ecaa0c692a8e1117db3
0625 13:51:59.259370 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.265770 (+ 6400us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.276459 (+ 10689us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.301429 (+ 24970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2283457}
I20250625 13:51:59.302289 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 329) took 2326 ms. Trace:
I20250625 13:51:59.302443 1165 rpcz_store.cc:276] 0625 13:51:56.975608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.975944 (+ 336us) service_pool.cc:225] Handling call
0625 13:51:59.302118 (+2326174us) ts_tablet_manager.cc:1145] Deleting tablet 9514bbbda0ba4329a18b0a42e635c6e5
0625 13:51:59.302128 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.302233 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2326001}
I20250625 13:51:59.302834 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d9bb7ebc40a14ecaa0c692a8e1117db3 (table default.Table_40de6f83d3af49f1a0bc517c4d0d875e [id=6e50ba5e257842e080a70caa1679952d]) successfully deleted
I20250625 13:51:59.302985 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 335) took 2065 ms. Trace:
W20250625 13:51:59.303256 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9514bbbda0ba4329a18b0a42e635c6e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9514bbbda0ba4329a18b0a42e635c6e5 already in progress: creating tablet
I20250625 13:51:59.303144 1165 rpcz_store.cc:276] 0625 13:51:57.237805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.238117 (+ 312us) service_pool.cc:225] Handling call
0625 13:51:59.302851 (+2064734us) ts_tablet_manager.cc:1145] Deleting tablet 5d7704b38144429b9c95675be84a4730
0625 13:51:59.302862 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.302937 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2064496}
I20250625 13:51:59.303683 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 336) took 2065 ms. Trace:
W20250625 13:51:59.303678 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5d7704b38144429b9c95675be84a4730 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d7704b38144429b9c95675be84a4730 already in progress: creating tablet
I20250625 13:51:59.303861 1165 rpcz_store.cc:276] 0625 13:51:57.237935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.238391 (+ 456us) service_pool.cc:225] Handling call
0625 13:51:59.303548 (+2065157us) ts_tablet_manager.cc:1145] Deleting tablet 63d01af6b320489fb99c467c11ed1dc0
0625 13:51:59.303558 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.303635 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2065001}
W20250625 13:51:59.304276 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 63d01af6b320489fb99c467c11ed1dc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63d01af6b320489fb99c467c11ed1dc0 already in progress: creating tablet
I20250625 13:51:59.304337 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 337) took 2066 ms. Trace:
I20250625 13:51:59.304473 1165 rpcz_store.cc:276] 0625 13:51:57.238054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.238577 (+ 523us) service_pool.cc:225] Handling call
0625 13:51:59.304204 (+2065627us) ts_tablet_manager.cc:1145] Deleting tablet 4fa801f7d98e4794b6ca439638f29e69
0625 13:51:59.304212 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.304285 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2065488}
W20250625 13:51:59.304935 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4fa801f7d98e4794b6ca439638f29e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fa801f7d98e4794b6ca439638f29e69 already in progress: creating tablet
I20250625 13:51:59.304976 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 338) took 2049 ms. Trace:
I20250625 13:51:59.305084 1165 rpcz_store.cc:276] 0625 13:51:57.255261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.255308 (+ 47us) service_pool.cc:225] Handling call
0625 13:51:59.304798 (+2049490us) ts_tablet_manager.cc:1145] Deleting tablet f73f420a4d7a438eadde3d6af64f0a11
0625 13:51:59.304806 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.304931 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049234}
I20250625 13:51:59.305621 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 339) took 2035 ms. Trace:
I20250625 13:51:59.306239 1165 rpcz_store.cc:276] 0625 13:51:57.269900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.269958 (+ 58us) service_pool.cc:225] Handling call
0625 13:51:59.305489 (+2035531us) ts_tablet_manager.cc:1145] Deleting tablet 70959ec681524518a7ec2c9ed4016ad1
0625 13:51:59.305498 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.305574 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035316}
W20250625 13:51:59.306399 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f73f420a4d7a438eadde3d6af64f0a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f73f420a4d7a438eadde3d6af64f0a11 already in progress: creating tablet
I20250625 13:51:59.306687 1165 tablet_replica.cc:331] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:59.306924 1165 raft_consensus.cc:2241] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.307147 1165 raft_consensus.cc:2270] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.307746 1165 ts_tablet_manager.cc:1905] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:51:59.307806 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 70959ec681524518a7ec2c9ed4016ad1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70959ec681524518a7ec2c9ed4016ad1 already in progress: creating tablet
I20250625 13:51:59.310578 1518 raft_consensus.cc:491] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.311043 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet d26820eaba98401da3a8b5348be21fc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.313205 880 tablet_service.cc:1515] Processing DeleteTablet for tablet d26820eaba98401da3a8b5348be21fc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.312757 1518 raft_consensus.cc:513] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.313898 1510 raft_consensus.cc:491] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.314093 1606 raft_consensus.cc:491] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.315109 1510 raft_consensus.cc:513] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:59.315673 880 tablet_service.cc:1468] Processing CreateTablet for tablet 45d68de75ac84f1a8138c38fd51547cf (DEFAULT_TABLE table=default.Table_9150143ab7394f1baf2fb087e8265dfa [id=f60ff5e9620e422d9bb3b03ec34e1f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.317461 1606 raft_consensus.cc:513] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:51:59.317574 983 consensus_peers.cc:487] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a21335a06ec04da29b3351554c75f03c. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:59.317682 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 0012e95ff4924723a6e6aaf6b85da438 (DEFAULT_TABLE table=default.Table_af0069799827448399dea82497b0b4d3 [id=c0d109f727f14a0eb8abd4ca7ff94dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.318099 1606 leader_election.cc:290] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:59.319082 851 leader_election.cc:343] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:59.319538 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.319909 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 17b172c7631048ceb97ccbea896f7f5b (DEFAULT_TABLE table=default.Table_93d4946c778542148fe4644b897607be [id=33e2c23d06724e51a7217dbe0780313d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.319927 896 tablet_service.cc:1468] Processing CreateTablet for tablet 0012e95ff4924723a6e6aaf6b85da438 (DEFAULT_TABLE table=default.Table_af0069799827448399dea82497b0b4d3 [id=c0d109f727f14a0eb8abd4ca7ff94dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.320112 1373 raft_consensus.cc:491] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.320282 1373 raft_consensus.cc:513] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.320874 1373 leader_election.cc:290] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:59.320851 882 tablet_service.cc:1468] Processing CreateTablet for tablet 2c1e362b69fe40c0b6472b56f8cb9bcb (DEFAULT_TABLE table=default.Table_0376b3d69a954c6881b4fd9d11e4b86b [id=0d141ebb7fc74195a5c05eeb73708142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.322104 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17b172c7631048ceb97ccbea896f7f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.323472 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93f449c80b604a6498f6fd1d1ba4770a" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:59.323611 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 2c1e362b69fe40c0b6472b56f8cb9bcb (DEFAULT_TABLE table=default.Table_0376b3d69a954c6881b4fd9d11e4b86b [id=0d141ebb7fc74195a5c05eeb73708142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.323976 1510 leader_election.cc:290] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:59.324162 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d68de75ac84f1a8138c38fd51547cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.325102 879 tablet_service.cc:1468] Processing CreateTablet for tablet 1310fee26fe642bf9919a2bd8e1c091f (DEFAULT_TABLE table=default.Table_f4aa1ee5e69c4dd6a8917a9a9900768f [id=ad66b451137740d1be88d600e320e218]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.325904 1166 tablet_bootstrap.cc:492] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.326054 1166 ts_tablet_manager.cc:1397] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.001s
I20250625 13:51:59.326084 895 tablet_service.cc:1515] Processing DeleteTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.326515 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0012e95ff4924723a6e6aaf6b85da438. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.326666 711 leader_election.cc:343] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.327021 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1e362b69fe40c0b6472b56f8cb9bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.326920 1166 raft_consensus.cc:357] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.327101 1166 raft_consensus.cc:383] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.327188 1166 raft_consensus.cc:738] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.327430 1166 consensus_queue.cc:260] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.327847 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 1310fee26fe642bf9919a2bd8e1c091f (DEFAULT_TABLE table=default.Table_f4aa1ee5e69c4dd6a8917a9a9900768f [id=ad66b451137740d1be88d600e320e218]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.327955 1166 ts_tablet_manager.cc:1428] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:59.328236 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1310fee26fe642bf9919a2bd8e1c091f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.328259 1166 tablet_bootstrap.cc:492] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:51:59.329031 985 leader_election.cc:343] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2c1a98dc40684954a602567cf3476879
I20250625 13:51:59.329946 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1e362b69fe40c0b6472b56f8cb9bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.333180 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1310fee26fe642bf9919a2bd8e1c091f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.333218 877 tablet_service.cc:1468] Processing CreateTablet for tablet cbf7e30cdff245208dacc6963fe88c7d (DEFAULT_TABLE table=default.Table_f2851488b3864c7b893b63c84369288d [id=69ed44196b63410cbf629e9ed2d928d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.333514 895 tablet_service.cc:1468] Processing CreateTablet for tablet 17b172c7631048ceb97ccbea896f7f5b (DEFAULT_TABLE table=default.Table_93d4946c778542148fe4644b897607be [id=33e2c23d06724e51a7217dbe0780313d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.334393 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17b172c7631048ceb97ccbea896f7f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.334636 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf7e30cdff245208dacc6963fe88c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.335618 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ecb9a98e402b4da8a9c2bd8317cef532
I20250625 13:51:59.336045 1510 raft_consensus.cc:491] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c1aa07ab6bc4b0089e46951240a37c9)
I20250625 13:51:59.336196 1510 raft_consensus.cc:513] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.336565 887 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.337620 1510 leader_election.cc:290] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:59.337635 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e3b3d5a2ab7042c2a2f18890363e7a9b (DEFAULT_TABLE table=default.Table_4145b0b039bc4d73a22b72a6340b02c5 [id=152af459c4be43638dbb84adb3a1b98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.338100 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b3d5a2ab7042c2a2f18890363e7a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.338470 887 tablet_service.cc:1468] Processing CreateTablet for tablet e3b3d5a2ab7042c2a2f18890363e7a9b (DEFAULT_TABLE table=default.Table_4145b0b039bc4d73a22b72a6340b02c5 [id=152af459c4be43638dbb84adb3a1b98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.338814 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b3d5a2ab7042c2a2f18890363e7a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.338824 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.341161 892 tablet_service.cc:1468] Processing CreateTablet for tablet bb648c45ea344fb3bac765c9e06db2b1 (DEFAULT_TABLE table=default.Table_e22c886829ef45be814505c0f1705376 [id=c6368a0cf1274d70ae05ebc9e565d8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.342083 1028 tablet_service.cc:1468] Processing CreateTablet for tablet bb648c45ea344fb3bac765c9e06db2b1 (DEFAULT_TABLE table=default.Table_e22c886829ef45be814505c0f1705376 [id=c6368a0cf1274d70ae05ebc9e565d8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.342927 1015 tablet_service.cc:1468] Processing CreateTablet for tablet cbf7e30cdff245208dacc6963fe88c7d (DEFAULT_TABLE table=default.Table_f2851488b3864c7b893b63c84369288d [id=69ed44196b63410cbf629e9ed2d928d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.343370 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf7e30cdff245208dacc6963fe88c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.345768 756 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.346181 753 tablet_service.cc:1468] Processing CreateTablet for tablet e3b3d5a2ab7042c2a2f18890363e7a9b (DEFAULT_TABLE table=default.Table_4145b0b039bc4d73a22b72a6340b02c5 [id=152af459c4be43638dbb84adb3a1b98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.346549 756 tablet_service.cc:1468] Processing CreateTablet for tablet bb648c45ea344fb3bac765c9e06db2b1 (DEFAULT_TABLE table=default.Table_e22c886829ef45be814505c0f1705376 [id=c6368a0cf1274d70ae05ebc9e565d8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.347055 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb648c45ea344fb3bac765c9e06db2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.347119 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 50786a4c25274ffb8f9910c5dafe3933 (table default.Table_55953048150049c490cd5b9346184cd0 [id=d7cbe338b2554a6e99b07325632aaa33]). Expected version 2 got 0
I20250625 13:51:59.347295 757 tablet_service.cc:1468] Processing CreateTablet for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 (DEFAULT_TABLE table=default.Table_0e7f3b83e2c7403eb46bc273da03cee8 [id=1286a09e25024ae58684376e9919eb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.347571 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b3d5a2ab7042c2a2f18890363e7a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.347772 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.347826 983 leader_election.cc:343] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 2c1a98dc40684954a602567cf3476879
I20250625 13:51:59.348253 983 leader_election.cc:304] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.350836 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_22678b3f1a3b4dd4afcafdc967bf2e93" } modify_external_catalogs: true
I20250625 13:51:59.351037 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_22678b3f1a3b4dd4afcafdc967bf2e93" } modify_external_catalogs: true
I20250625 13:51:59.347872 1166 tablet_bootstrap.cc:654] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.355818 1373 raft_consensus.cc:491] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.355980 1373 raft_consensus.cc:513] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.356662 1373 leader_election.cc:290] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:59.356688 985 leader_election.cc:343] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4b549bc8cead4af4a72f41f51655602c
I20250625 13:51:59.356799 748 tablet_service.cc:1468] Processing CreateTablet for tablet 17b172c7631048ceb97ccbea896f7f5b (DEFAULT_TABLE table=default.Table_93d4946c778542148fe4644b897607be [id=33e2c23d06724e51a7217dbe0780313d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.348220 1165 ts_tablet_manager.cc:1918] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.357777 1165 log.cc:1199] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/07e281786e6e4a5d93b94b85b486655d
I20250625 13:51:59.358143 1165 ts_tablet_manager.cc:1939] T 07e281786e6e4a5d93b94b85b486655d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.358069 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_41e093f1c53c404fb4743c97d30f7f31" } new_table_name: "default.Table_478b0037c80e40048d4f8ebc92748f34" modify_external_catalogs: true
I20250625 13:51:59.358453 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17b172c7631048ceb97ccbea896f7f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.359099 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 341) took 2071 ms. Trace:
I20250625 13:51:59.359298 1165 rpcz_store.cc:276] 0625 13:51:57.287234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.287282 (+ 48us) service_pool.cc:225] Handling call
0625 13:51:59.306615 (+2019333us) ts_tablet_manager.cc:1145] Deleting tablet 07e281786e6e4a5d93b94b85b486655d
0625 13:51:59.306623 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.312540 (+ 5917us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.348080 (+ 35540us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.359054 (+ 10974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019036}
I20250625 13:51:59.359880 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 340) took 2075 ms. Trace:
I20250625 13:51:59.360035 1606 raft_consensus.cc:2747] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:59.360533 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0012e95ff4924723a6e6aaf6b85da438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.360863 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c1c9eece77144878a55acc01667331a9" } new_table_name: "default.Table_fe321782d15943acaf71e83adc22ab28" modify_external_catalogs: true
I20250625 13:51:59.361212 1165 rpcz_store.cc:276] 0625 13:51:57.284393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.303359 (+ 18966us) service_pool.cc:225] Handling call
0625 13:51:59.359735 (+2056376us) ts_tablet_manager.cc:1145] Deleting tablet 87f5b36ddb474ed6903c14876b236ba8
0625 13:51:59.359744 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.359843 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2056105}
I20250625 13:51:59.361713 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 353) took 1864 ms. Trace:
I20250625 13:51:59.361824 1165 rpcz_store.cc:276] 0625 13:51:57.497396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.498102 (+ 706us) service_pool.cc:225] Handling call
0625 13:51:59.361578 (+1863476us) ts_tablet_manager.cc:1145] Deleting tablet ac4d368347444846880954ced9651404
0625 13:51:59.361586 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.361683 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863210}
I20250625 13:51:59.361919 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 07e281786e6e4a5d93b94b85b486655d (table default.Table_fb311c42df024421b325596e0cb7a778 [id=7b67603755e7428496524ee716b435e9]) successfully deleted
W20250625 13:51:59.362262 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ac4d368347444846880954ced9651404 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac4d368347444846880954ced9651404 already in progress: creating tablet
I20250625 13:51:59.363620 743 tablet_service.cc:1468] Processing CreateTablet for tablet cbf7e30cdff245208dacc6963fe88c7d (DEFAULT_TABLE table=default.Table_f2851488b3864c7b893b63c84369288d [id=69ed44196b63410cbf629e9ed2d928d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.363901 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 430c3bed360b4f32855a8adb99076608
I20250625 13:51:59.364004 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf7e30cdff245208dacc6963fe88c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.364369 593 catalog_manager.cc:6652] ReplaceTablet: tablet 430c3bed360b4f32855a8adb99076608 deleted and replaced by tablet 0f63e9037bd74f1eb39bd85aa820ec35
I20250625 13:51:59.364809 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 354) took 1866 ms. Trace:
I20250625 13:51:59.364920 1165 rpcz_store.cc:276] 0625 13:51:57.498071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.498134 (+ 63us) service_pool.cc:225] Handling call
0625 13:51:59.364667 (+1866533us) ts_tablet_manager.cc:1145] Deleting tablet a22ade8cefac4140b08882c311ea5181
0625 13:51:59.364677 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.364766 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866277}
I20250625 13:51:59.365216 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 430c3bed360b4f32855a8adb99076608 with delete_type TABLET_DATA_DELETED (replaced by tablet 0f63e9037bd74f1eb39bd85aa820ec35) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.367658 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 355) took 1863 ms. Trace:
I20250625 13:51:59.367707 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 50786a4c25274ffb8f9910c5dafe3933 (table default.Table_55953048150049c490cd5b9346184cd0 [id=d7cbe338b2554a6e99b07325632aaa33]). Expected version 2 got 0
I20250625 13:51:59.367780 1165 rpcz_store.cc:276] 0625 13:51:57.504187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.504247 (+ 60us) service_pool.cc:225] Handling call
0625 13:51:59.367519 (+1863272us) ts_tablet_manager.cc:1145] Deleting tablet 27092991fcad475fabc77d1f8bab0582
0625 13:51:59.367528 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.367618 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863009}
I20250625 13:51:59.369089 1165 tablet_replica.cc:331] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:59.369354 1165 raft_consensus.cc:2241] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:59.369421 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a22ade8cefac4140b08882c311ea5181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a22ade8cefac4140b08882c311ea5181 already in progress: creating tablet
I20250625 13:51:59.369581 1165 raft_consensus.cc:2270] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:51:59.369692 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 27092991fcad475fabc77d1f8bab0582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27092991fcad475fabc77d1f8bab0582 already in progress: creating tablet
I20250625 13:51:59.370381 1165 ts_tablet_manager.cc:1905] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.377013 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_55953048150049c490cd5b9346184cd0" } new_table_name: "default.Table_ebb1cd6aa381410a82cac38e7197353c" modify_external_catalogs: true
I20250625 13:51:59.377514 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 35aedb3b590d4917bac8fe0962d68fb3 (DEFAULT_TABLE table=default.Table_ad21d2aa5ffb4443bb2d0b96596545c3 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.378551 756 tablet_service.cc:1515] Processing DeleteTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.379825 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ba042527b71242e8834fd66124edd5a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.380326 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba042527b71242e8834fd66124edd5a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.381609 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1f8b426bfe4f426a9840a85d92052857"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.382082 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f8b426bfe4f426a9840a85d92052857 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.383723 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_98ae9c7310a24e8d988b919a5ec496e7" } modify_external_catalogs: true
I20250625 13:51:59.383903 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_98ae9c7310a24e8d988b919a5ec496e7" } modify_external_catalogs: true
I20250625 13:51:59.384423 1518 leader_election.cc:290] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:59.384685 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "583ee2f2c66841939c93104efc36facb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:59.385505 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_66186ab6312345c1a94ca39ad501c296"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.385991 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66186ab6312345c1a94ca39ad501c296 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.386596 756 tablet_service.cc:1468] Processing CreateTablet for tablet 35aedb3b590d4917bac8fe0962d68fb3 (DEFAULT_TABLE table=default.Table_ad21d2aa5ffb4443bb2d0b96596545c3 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.387091 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35aedb3b590d4917bac8fe0962d68fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.387508 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_aba4565f0af54d4b984435978a9c2821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.387989 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aba4565f0af54d4b984435978a9c2821 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.388239 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41a7862601f04fd7a938f7e5d5a4c0f9 consensus state has no leader
I20250625 13:51:59.387985 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6f5780d88dea41e7b855b41f12e96c52
I20250625 13:51:59.391748 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ff33e931cb1c4e44a12f7dcb8ddad00c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.392655 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f5780d88dea41e7b855b41f12e96c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:59.392736 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff33e931cb1c4e44a12f7dcb8ddad00c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.393929 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 consensus state has no leader
I20250625 13:51:59.386596 753 tablet_service.cc:1468] Processing CreateTablet for tablet e73bea55af9e4890bf7070b488313c3f (DEFAULT_TABLE table=default.Table_528b80e737c9462194cd9b50d91b7db9 [id=171d9555defd4d1ba8634fb0edfb55d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.396432 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73bea55af9e4890bf7070b488313c3f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.396677 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50786a4c25274ffb8f9910c5dafe3933 consensus state has no leader
I20250625 13:51:59.400493 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_12771e585c3348049cc0ab4bf2a95935" } modify_external_catalogs: true
W20250625 13:51:59.400596 851 leader_election.cc:343] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7108932fa8a241819ca90674a64b285e
I20250625 13:51:59.400725 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_12771e585c3348049cc0ab4bf2a95935" } modify_external_catalogs: true
I20250625 13:51:59.401114 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_01f1a91fcc43403db4e9c2092f12a888"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.401690 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01f1a91fcc43403db4e9c2092f12a888 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.403487 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 57642.2us
I20250625 13:51:59.404057 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e73bea55af9e4890bf7070b488313c3f (DEFAULT_TABLE table=default.Table_528b80e737c9462194cd9b50d91b7db9 [id=171d9555defd4d1ba8634fb0edfb55d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.404428 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93f449c80b604a6498f6fd1d1ba4770a" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:59.405455 985 consensus_peers.cc:487] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67441795e22b445a90c88d76aa519ea5. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:59.405596 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73bea55af9e4890bf7070b488313c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.410760 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 430c3bed360b4f32855a8adb99076608 with delete_type TABLET_DATA_DELETED (replaced by tablet 0f63e9037bd74f1eb39bd85aa820ec35) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.410782 884 tablet_service.cc:1468] Processing CreateTablet for tablet 35aedb3b590d4917bac8fe0962d68fb3 (DEFAULT_TABLE table=default.Table_ad21d2aa5ffb4443bb2d0b96596545c3 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.416322 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f5780d88dea41e7b855b41f12e96c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.416551 1030 tablet_service.cc:1468] Processing CreateTablet for tablet efc9536a716f4b9f9fc4af1637b3cd80 (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:51:59.418861 983 leader_election.cc:343] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 4b549bc8cead4af4a72f41f51655602c
I20250625 13:51:59.419008 983 leader_election.cc:304] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.419512 1518 raft_consensus.cc:2747] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:59.420984 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 (DEFAULT_TABLE table=default.Table_0e7f3b83e2c7403eb46bc273da03cee8 [id=1286a09e25024ae58684376e9919eb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.421777 1170 tablet_bootstrap.cc:492] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:59.422466 1170 ts_tablet_manager.cc:1397] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.328s user 0.005s sys 0.011s
I20250625 13:51:59.423849 1170 raft_consensus.cc:357] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.425455 1170 raft_consensus.cc:383] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.425560 1170 raft_consensus.cc:738] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.426054 1170 consensus_queue.cc:260] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.426573 1170 ts_tablet_manager.cc:1428] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:51:59.426898 1170 tablet_bootstrap.cc:492] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:59.189252 1473 rpcz_store.cc:276] 0625 13:51:57.873249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.882828 (+ 9579us) service_pool.cc:225] Handling call
0625 13:51:58.437059 (+554231us) raft_consensus.cc:1469] Updating replica for 432 ops
0625 13:51:58.462725 (+ 25666us) raft_consensus.cc:1532] Early marking committed up to index 14
0625 13:51:58.557501 (+ 94776us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:51:58.557514 (+ 13us) write_op.cc:454] Released schema lock
0625 13:51:58.559243 (+ 1729us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:51:58.559248 (+ 5us) write_op.cc:454] Released schema lock
0625 13:51:58.559423 (+ 175us) raft_consensus.cc:1537] Triggering prepare for 432 ops
0625 13:51:58.561400 (+ 1977us) write_op.cc:270] Start()
0625 13:51:58.561450 (+ 50us) write_op.cc:276] Timestamp: P: 1750859511292951 usec, L: 0
0625 13:51:58.564105 (+ 2655us) write_op.cc:270] Start()
0625 13:51:58.564132 (+ 27us) write_op.cc:276] Timestamp: P: 1750859511295317 usec, L: 0
0625 13:51:58.564848 (+ 716us) write_op.cc:270] Start()
0625 13:51:58.564870 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511296324 usec, L: 0
0625 13:51:58.565789 (+ 919us) write_op.cc:270] Start()
0625 13:51:58.565812 (+ 23us) write_op.cc:276] Timestamp: P: 1750859511296896 usec, L: 0
0625 13:51:58.566362 (+ 550us) write_op.cc:270] Start()
0625 13:51:58.566384 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511298557 usec, L: 0
0625 13:51:58.566920 (+ 536us) write_op.cc:270] Start()
0625 13:51:58.567318 (+ 398us) write_op.cc:276] Timestamp: P: 1750859511299064 usec, L: 0
0625 13:51:58.567986 (+ 668us) write_op.cc:270] Start()
0625 13:51:58.568015 (+ 29us) write_op.cc:276] Timestamp: P: 1750859511299596 usec, L: 0
0625 13:51:58.569009 (+ 994us) write_op.cc:270] Start()
0625 13:51:58.569031 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511311888 usec, L: 0
0625 13:51:58.573625 (+ 4594us) write_op.cc:270] Start()
0625 13:51:58.573658 (+ 33us) write_op.cc:276] Timestamp: P: 1750859511313484 usec, L: 0
0625 13:51:58.573829 (+ 171us) write_op.cc:270] Start()
0625 13:51:58.573852 (+ 23us) write_op.cc:276] Timestamp: P: 1750859511314270 usec, L: 0
0625 13:51:58.574005 (+ 153us) write_op.cc:270] Start()
0625 13:51:58.574028 (+ 23us) write_op.cc:276] Timestamp: P: 1750859511322152 usec, L: 0
0625 13:51:58.574185 (+ 157us) write_op.cc:270] Start()
0625 13:51:58.574208 (+ 23us) write_op.cc:276] Timestamp: P: 1750859511472680 usec, L: 0
0625 13:51:58.574360 (+ 152us) write_op.cc:270] Start()
0625 13:51:58.574383 (+ 23us) write_op.cc:276] Timestamp: P: 1750859511475103 usec, L: 0
0625 13:51:58.574526 (+ 143us) write_op.cc:270] Start()
0625 13:51:58.574550 (+ 24us) write_op.cc:276] Timestamp: P: 1750859511481008 usec, L: 0
0625 13:51:58.574960 (+ 410us) write_op.cc:270] Start()
0625 13:51:58.574985 (+ 25us) write_op.cc:276] Timestamp: P: 1750859511483626 usec, L: 0
0625 13:51:58.575122 (+ 137us) write_op.cc:270] Start()
0625 13:51:58.575144 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511484191 usec, L: 0
0625 13:51:58.575291 (+ 147us) write_op.cc:270] Start()
0625 13:51:58.575363 (+ 72us) write_op.cc:276] Timestamp: P: 1750859511491545 usec, L: 0
0625 13:51:58.575751 (+ 388us) write_op.cc:270] Start()
0625 13:51:58.575771 (+ 20us) write_op.cc:276] Timestamp: P: 1750859511498665 usec, L: 0
0625 13:51:58.576076 (+ 305us) write_op.cc:270] Start()
0625 13:51:58.576097 (+ 21us) write_op.cc:276] Timestamp: P: 1750859511506315 usec, L: 0
0625 13:51:58.576976 (+ 879us) write_op.cc:270] Start()
0625 13:51:58.576998 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511506822 usec, L: 0
0625 13:51:58.577124 (+ 126us) write_op.cc:270] Start()
0625 13:51:58.577141 (+ 17us) write_op.cc:276] Timestamp: P: 1750859511507268 usec, L: 0
0625 13:51:58.577273 (+ 132us) write_op.cc:270] Start()
0625 13:51:58.577316 (+ 43us) write_op.cc:276] Timestamp: P: 1750859511525902 usec, L: 0
0625 13:51:58.577431 (+ 115us) write_op.cc:270] Start()
0625 13:51:58.577448 (+ 17us) write_op.cc:276] Timestamp: P: 1750859511526733 usec, L: 0
0625 13:51:58.577566 (+ 118us) write_op.cc:270] Start()
0625 13:51:58.577583 (+ 17us) write_op.cc:276] Timestamp: P: 1750859511527403 usec, L: 0
0625 13:51:58.578857 (+ 1274us) write_op.cc:270] Start()
0625 13:51:58.578881 (+ 24us) write_op.cc:276] Timestamp: P: 1750859511535038 usec, L: 0
0625 13:51:58.579011 (+ 130us) write_op.cc:270] Start()
0625 13:51:58.579028 (+ 17us) write_op.cc:276] Timestamp: P: 1750859511547360 usec, L: 0
0625 13:51:58.579135 (+ 107us) write_op.cc:270] Start()
0625 13:51:58.579152 (+ 17us) write_op.cc:276] Timestamp: P: 1750859511547991 usec, L: 0
0625 13:51:58.579264 (+ 112us) write_op.cc:270] Start()
0625 13:51:58.579279 (+ 15us) write_op.cc:276] Timestamp: P: 1750859511548534 usec, L: 0
0625 13:51:58.579394 (+ 115us) write_op.cc:270] Start()
0625 13:51:58.579410 (+ 16us) write_op.cc:276] Timestamp: P: 1750859511582604 usec, L: 0
0625 13:51:58.579524 (+ 114us) write_op.cc:270] Start()
0625 13:51:58.579541 (+ 17us) write_op.cc:276] Timestamp: P: 1750859511588125 usec, L: 0
0625 13:51:58.603511 (+ 23970us) write_op.cc:270] Start()
0625 13:51:58.603703 (+ 192us) write_op.cc:276] Timestamp: P: 1750859511596432 usec, L: 0
0625 13:51:58.603978 (+ 275us) write_op.cc:270] Start()
0625 13:51:58.604006 (+ 28us) write_op.cc:276] Timestamp: P: 1750859511650310 usec, L: 0
0625 13:51:58.604198 (+ 192us) write_op.cc:270] Start()
0625 13:51:58.604221 (+ 23us) write_op.cc:276] Timestamp: P: 1750859511661525 usec, L: 0
0625 13:51:58.604374 (+ 153us) write_op.cc:270] Start()
0625 13:51:58.604395 (+ 21us) write_op.cc:276] Timestamp: P: 1750859511673170 usec, L: 0
0625 13:51:58.604536 (+ 141us) write_op.cc:270] Start()
0625 13:51:58.604558 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511685369 usec, L: 0
0625 13:51:58.615852 (+ 11294us) write_op.cc:270] Start()
0625 13:51:58.615889 (+ 37us) write_op.cc:276] Timestamp: P: 1750859511686016 usec, L: 0
0625 13:51:58.616079 (+ 190us) write_op.cc:270] Start()
0625 13:51:58.616112 (+ 33us) write_op.cc:276] Timestamp: P: 1750859511686532 usec, L: 0
0625 13:51:58.616265 (+ 153us) write_op.cc:270] Start()
0625 13:51:58.616284 (+ 19us) write_op.cc:276] Timestamp: P: 1750859511745470 usec, L: 0
0625 13:51:58.616467 (+ 183us) write_op.cc:270] Start()
0625 13:51:58.616492 (+ 25us) write_op.cc:276] Timestamp: P: 1750859511746324 usec, L: 0
0625 13:51:58.616661 (+ 169us) write_op.cc:270] Start()
0625 13:51:58.616688 (+ 27us) write_op.cc:276] Timestamp: P: 1750859511791198 usec, L: 0
0625 13:51:58.616824 (+ 136us) write_op.cc:270] Start()
0625 13:51:58.616844 (+ 20us) write_op.cc:276] Timestamp: P: 1750859511798659 usec, L: 0
0625 13:51:58.617117 (+ 273us) write_op.cc:270] Start()
0625 13:51:58.617150 (+ 33us) write_op.cc:276] Timestamp: P: 1750859511805262 usec, L: 0
0625 13:51:58.617298 (+ 148us) write_op.cc:270] Start()
0625 13:51:58.617324 (+ 26us) write_op.cc:276] Timestamp: P: 1750859511845417 usec, L: 0
0625 13:51:58.617484 (+ 160us) write_op.cc:270] Start()
0625 13:51:58.617505 (+ 21us) write_op.cc:276] Timestamp: P: 1750859511855213 usec, L: 0
0625 13:51:58.617680 (+ 175us) write_op.cc:270] Start()
0625 13:51:58.617707 (+ 27us) write_op.cc:276] Timestamp: P: 1750859511904348 usec, L: 0
0625 13:51:58.617873 (+ 166us) write_op.cc:270] Start()
0625 13:51:58.617897 (+ 24us) write_op.cc:276] Timestamp: P: 1750859511931610 usec, L: 0
0625 13:51:58.618051 (+ 154us) write_op.cc:270] Start()
0625 13:51:58.618075 (+ 24us) write_op.cc:276] Timestamp: P: 1750859511940420 usec, L: 0
0625 13:51:58.618224 (+ 149us) write_op.cc:270] Start()
0625 13:51:58.618246 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511950729 usec, L: 0
0625 13:51:58.618388 (+ 142us) write_op.cc:270] Start()
0625 13:51:58.618408 (+ 20us) write_op.cc:276] Timestamp: P: 1750859511959916 usec, L: 0
0625 13:51:58.618564 (+ 156us) write_op.cc:270] Start()
0625 13:51:58.618586 (+ 22us) write_op.cc:276] Timestamp: P: 1750859511965189 usec, L: 0
0625 13:51:58.618739 (+ 153us) write_op.cc:270] Start()
0625 13:51:58.618762 (+ 23us) write_op.cc:276] Timestamp: P: 1750859512012998 usec, L: 0
0625 13:51:58.618884 (+ 122us) write_op.cc:270] Start()
0625 13:51:58.618905 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512040540 usec, L: 0
0625 13:51:58.619043 (+ 138us) write_op.cc:270] Start()
0625 13:51:58.619067 (+ 24us) write_op.cc:276] Timestamp: P: 1750859512061726 usec, L: 0
0625 13:51:58.619213 (+ 146us) write_op.cc:270] Start()
0625 13:51:58.619234 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512072974 usec, L: 0
0625 13:51:58.620209 (+ 975us) write_op.cc:270] Start()
0625 13:51:58.620238 (+ 29us) write_op.cc:276] Timestamp: P: 1750859512085613 usec, L: 0
0625 13:51:58.620393 (+ 155us) write_op.cc:270] Start()
0625 13:51:58.620414 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512094906 usec, L: 0
0625 13:51:58.620570 (+ 156us) write_op.cc:270] Start()
0625 13:51:58.620602 (+ 32us) write_op.cc:276] Timestamp: P: 1750859512105110 usec, L: 0
0625 13:51:58.620768 (+ 166us) write_op.cc:270] Start()
0625 13:51:58.620789 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512105842 usec, L: 0
0625 13:51:58.620924 (+ 135us) write_op.cc:270] Start()
0625 13:51:58.620944 (+ 20us) write_op.cc:276] Timestamp: P: 1750859512106692 usec, L: 0
0625 13:51:58.621127 (+ 183us) write_op.cc:270] Start()
0625 13:51:58.621156 (+ 29us) write_op.cc:276] Timestamp: P: 1750859512107413 usec, L: 0
0625 13:51:58.621314 (+ 158us) write_op.cc:270] Start()
0625 13:51:58.621336 (+ 22us) write_op.cc:276] Timestamp: P: 1750859512186016 usec, L: 0
0625 13:51:58.621474 (+ 138us) write_op.cc:270] Start()
0625 13:51:58.621497 (+ 23us) write_op.cc:276] Timestamp: P: 1750859512186690 usec, L: 0
0625 13:51:58.621643 (+ 146us) write_op.cc:270] Start()
0625 13:51:58.621666 (+ 23us) write_op.cc:276] Timestamp: P: 1750859512187809 usec, L: 0
0625 13:51:58.621864 (+ 198us) write_op.cc:270] Start()
0625 13:51:58.621888 (+ 24us) write_op.cc:276] Timestamp: P: 1750859512188617 usec, L: 0
0625 13:51:58.622075 (+ 187us) write_op.cc:270] Start()
0625 13:51:58.622099 (+ 24us) write_op.cc:276] Timestamp: P: 1750859512189405 usec, L: 0
0625 13:51:58.622239 (+ 140us) write_op.cc:270] Start()
0625 13:51:58.622260 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512190152 usec, L: 0
0625 13:51:58.622399 (+ 139us) write_op.cc:270] Start()
0625 13:51:58.622421 (+ 22us) write_op.cc:276] Timestamp: P: 1750859512212633 usec, L: 0
0625 13:51:58.622554 (+ 133us) write_op.cc:270] Start()
0625 13:51:58.622569 (+ 15us) write_op.cc:276] Timestamp: P: 1750859512213509 usec, L: 0
0625 13:51:58.625406 (+ 2837us) write_op.cc:270] Start()
0625 13:51:58.625436 (+ 30us) write_op.cc:276] Timestamp: P: 1750859512225730 usec, L: 0
0625 13:51:58.625614 (+ 178us) write_op.cc:270] Start()
0625 13:51:58.625639 (+ 25us) write_op.cc:276] Timestamp: P: 1750859512227462 usec, L: 0
0625 13:51:58.625778 (+ 139us) write_op.cc:270] Start()
0625 13:51:58.625796 (+ 18us) write_op.cc:276] Timestamp: P: 1750859512234808 usec, L: 0
0625 13:51:58.625912 (+ 116us) write_op.cc:270] Start()
0625 13:51:58.625928 (+ 16us) write_op.cc:276] Timestamp: P: 1750859512244239 usec, L: 0
0625 13:51:58.626057 (+ 129us) write_op.cc:270] Start()
0625 13:51:58.626083 (+ 26us) write_op.cc:276] Timestamp: P: 1750859512302310 usec, L: 0
0625 13:51:58.626221 (+ 138us) write_op.cc:270] Start()
0625 13:51:58.626240 (+ 19us) write_op.cc:276] Timestamp: P: 1750859512329233 usec, L: 0
0625 13:51:58.633570 (+ 7330us) write_op.cc:270] Start()
0625 13:51:58.633618 (+ 48us) write_op.cc:276] Timestamp: P: 1750859512336444 usec, L: 0
0625 13:51:58.633886 (+ 268us) write_op.cc:270] Start()
0625 13:51:58.633922 (+ 36us) write_op.cc:276] Timestamp: P: 1750859512351043 usec, L: 0
0625 13:51:58.634197 (+ 275us) write_op.cc:270] Start()
0625 13:51:58.634226 (+ 29us) write_op.cc:276] Timestamp: P: 1750859512363159 usec, L: 0
0625 13:51:58.634387 (+ 161us) write_op.cc:270] Start()
0625 13:51:58.634434 (+ 47us) write_op.cc:276] Timestamp: P: 1750859512363880 usec, L: 0
0625 13:51:58.634847 (+ 413us) write_op.cc:270] Start()
0625 13:51:58.634950 (+ 103us) write_op.cc:276] Timestamp: P: 1750859512364649 usec, L: 0
0625 13:51:58.635141 (+ 191us) write_op.cc:270] Start()
0625 13:51:58.635159 (+ 18us) write_op.cc:276] Timestamp: P: 1750859512406845 usec, L: 0
0625 13:51:58.635321 (+ 162us) write_op.cc:270] Start()
0625 13:51:58.635348 (+ 27us) write_op.cc:276] Timestamp: P: 1750859512408327 usec, L: 0
0625 13:51:58.635502 (+ 154us) write_op.cc:270] Start()
0625 13:51:58.635522 (+ 20us) write_op.cc:276] Timestamp: P: 1750859512410483 usec, L: 0
0625 13:51:58.635653 (+ 131us) write_op.cc:270] Start()
0625 13:51:58.635670 (+ 17us) write_op.cc:276] Timestamp: P: 1750859512420121 usec, L: 0
0625 13:51:58.635795 (+ 125us) write_op.cc:270] Start()
0625 13:51:58.635818 (+ 23us) write_op.cc:276] Timestamp: P: 1750859512443327 usec, L: 0
0625 13:51:58.635931 (+ 113us) write_op.cc:270] Start()
0625 13:51:58.635948 (+ 17us) write_op.cc:276] Timestamp: P: 1750859512451791 usec, L: 0
0625 13:51:58.636059 (+ 111us) write_op.cc:270] Start()
0625 13:51:58.636073 (+ 14us) write_op.cc:276] Timestamp: P: 1750859512485644 usec, L: 0
0625 13:51:58.636173 (+ 100us) write_op.cc:270] Start()
0625 13:51:58.636188 (+ 15us) write_op.cc:276] Timestamp: P: 1750859512527515 usec, L: 0
0625 13:51:58.636306 (+ 118us) write_op.cc:270] Start()
0625 13:51:58.636324 (+ 18us) write_op.cc:276] Timestamp: P: 1750859512535357 usec, L: 0
0625 13:51:58.636451 (+ 127us) write_op.cc:270] Start()
0625 13:51:58.636470 (+ 19us) write_op.cc:276] Timestamp: P: 1750859512549626 usec, L: 0
0625 13:51:58.636622 (+ 152us) write_op.cc:270] Start()
0625 13:51:58.636641 (+ 19us) write_op.cc:276] Timestamp: P: 1750859512557385 usec, L: 0
0625 13:51:58.636772 (+ 131us) write_op.cc:270] Start()
0625 13:51:58.636792 (+ 20us) write_op.cc:276] Timestamp: P: 1750859512572755 usec, L: 0
0625 13:51:58.636932 (+ 140us) write_op.cc:270] Start()
0625 13:51:58.636950 (+ 18us) write_op.cc:276] Timestamp: P: 1750859512583334 usec, L: 0
0625 13:51:58.637085 (+ 135us) write_op.cc:270] Start()
0625 13:51:58.637130 (+ 45us) write_op.cc:276] Timestamp: P: 1750859512585176 usec, L: 0
0625 13:51:58.637333 (+ 203us) write_op.cc:270] Start()
0625 13:51:58.637359 (+ 26us) write_op.cc:276] Timestamp: P: 1750859512622265 usec, L: 0
0625 13:51:58.637510 (+ 151us) write_op.cc:270] Start()
0625 13:51:58.637529 (+ 19us) write_op.cc:276] Timestamp: P: 1750859512626726 usec, L: 0
0625 13:51:58.647373 (+ 9844us) write_op.cc:270] Start()
0625 13:51:58.647423 (+ 50us) write_op.cc:276] Timestamp: P: 1750859512672699 usec, L: 0
0625 13:51:58.647721 (+ 298us) write_op.cc:270] Start()
0625 13:51:58.647750 (+ 29us) write_op.cc:276] Timestamp: P: 1750859512673833 usec, L: 0
0625 13:51:58.647907 (+ 157us) write_op.cc:270] Start()
0625 13:51:58.647926 (+ 19us) write_op.cc:276] Timestamp: P: 1750859512674882 usec, L: 0
0625 13:51:58.648034 (+ 108us) write_op.cc:270] Start()
0625 13:51:58.648051 (+ 17us) write_op.cc:276] Timestamp: P: 1750859512676494 usec, L: 0
0625 13:51:58.648165 (+ 114us) write_op.cc:270] Start()
0625 13:51:58.648501 (+ 336us) write_op.cc:276] Timestamp: P: 1750859512678038 usec, L: 0
0625 13:51:58.648691 (+ 190us) write_op.cc:270] Start()
0625 13:51:58.648714 (+ 23us) write_op.cc:276] Timestamp: P: 1750859512679063 usec, L: 0
0625 13:51:58.648844 (+ 130us) write_op.cc:270] Start()
0625 13:51:58.648872 (+ 28us) write_op.cc:276] Timestamp: P: 1750859512704683 usec, L: 0
0625 13:51:58.649014 (+ 142us) write_op.cc:270] Start()
0625 13:51:58.649035 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512711985 usec, L: 0
0625 13:51:58.649172 (+ 137us) write_op.cc:270] Start()
0625 13:51:58.649195 (+ 23us) write_op.cc:276] Timestamp: P: 1750859512733630 usec, L: 0
0625 13:51:58.649340 (+ 145us) write_op.cc:270] Start()
0625 13:51:58.649362 (+ 22us) write_op.cc:276] Timestamp: P: 1750859512736873 usec, L: 0
0625 13:51:58.649814 (+ 452us) write_op.cc:270] Start()
0625 13:51:58.649839 (+ 25us) write_op.cc:276] Timestamp: P: 1750859512740272 usec, L: 0
0625 13:51:58.649979 (+ 140us) write_op.cc:270] Start()
0625 13:51:58.650002 (+ 23us) write_op.cc:276] Timestamp: P: 1750859512741300 usec, L: 0
0625 13:51:58.650169 (+ 167us) write_op.cc:270] Start()
0625 13:51:58.650193 (+ 24us) write_op.cc:276] Timestamp: P: 1750859512764024 usec, L: 0
0625 13:51:58.650343 (+ 150us) write_op.cc:270] Start()
0625 13:51:58.650364 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512764683 usec, L: 0
0625 13:51:58.650485 (+ 121us) write_op.cc:270] Start()
0625 13:51:58.650504 (+ 19us) write_op.cc:276] Timestamp: P: 1750859512770794 usec, L: 0
0625 13:51:58.650650 (+ 146us) write_op.cc:270] Start()
0625 13:51:58.650669 (+ 19us) write_op.cc:276] Timestamp: P: 1750859512787061 usec, L: 0
0625 13:51:58.650802 (+ 133us) write_op.cc:270] Start()
0625 13:51:58.650823 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512809061 usec, L: 0
0625 13:51:58.650946 (+ 123us) write_op.cc:270] Start()
0625 13:51:58.650967 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512810000 usec, L: 0
0625 13:51:58.651109 (+ 142us) write_op.cc:270] Start()
0625 13:51:58.651129 (+ 20us) write_op.cc:276] Timestamp: P: 1750859512821814 usec, L: 0
0625 13:51:58.651270 (+ 141us) write_op.cc:270] Start()
0625 13:51:58.651287 (+ 17us) write_op.cc:276] Timestamp: P: 1750859512823619 usec, L: 0
0625 13:51:58.651753 (+ 466us) write_op.cc:270] Start()
0625 13:51:58.651774 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512887711 usec, L: 0
0625 13:51:58.651933 (+ 159us) write_op.cc:270] Start()
0625 13:51:58.651950 (+ 17us) write_op.cc:276] Timestamp: P: 1750859512890572 usec, L: 0
0625 13:51:58.652803 (+ 853us) write_op.cc:270] Start()
0625 13:51:58.652824 (+ 21us) write_op.cc:276] Timestamp: P: 1750859512929886 usec, L: 0
0625 13:51:58.652960 (+ 136us) write_op.cc:270] Start()
0625 13:51:58.652977 (+ 17us) write_op.cc:276] Timestamp: P: 1750859512938096 usec, L: 0
0625 13:51:58.653077 (+ 100us) write_op.cc:270] Start()
0625 13:51:58.653093 (+ 16us) write_op.cc:276] Timestamp: P: 1750859512944042 usec, L: 0
0625 13:51:58.653185 (+ 92us) write_op.cc:270] Start()
0625 13:51:58.653201 (+ 16us) write_op.cc:276] Timestamp: P: 1750859512958191 usec, L: 0
0625 13:51:58.653323 (+ 122us) write_op.cc:270] Start()
0625 13:51:58.653340 (+ 17us) write_op.cc:276] Timestamp: P: 1750859512958867 usec, L: 0
0625 13:51:58.653452 (+ 112us) write_op.cc:270] Start()
0625 13:51:58.653468 (+ 16us) write_op.cc:276] Timestamp: P: 1750859513025267 usec, L: 0
0625 13:51:58.653627 (+ 159us) write_op.cc:270] Start()
0625 13:51:58.653645 (+ 18us) write_op.cc:276] Timestamp: P: 1750859513028910 usec, L: 0
0625 13:51:58.653747 (+ 102us) write_op.cc:270] Start()
0625 13:51:58.653761 (+ 14us) write_op.cc:276] Timestamp: P: 1750859513032028 usec, L: 0
0625 13:51:58.653850 (+ 89us) write_op.cc:270] Start()
0625 13:51:58.653865 (+ 15us) write_op.cc:276] Timestamp: P: 1750859513058005 usec, L: 0
0625 13:51:58.653989 (+ 124us) write_op.cc:270] Start()
0625 13:51:58.654008 (+ 19us) write_op.cc:276] Timestamp: P: 1750859513072717 usec, L: 0
0625 13:51:58.654207 (+ 199us) write_op.cc:270] Start()
0625 13:51:58.654236 (+ 29us) write_op.cc:276] Timestamp: P: 1750859513089772 usec, L: 0
0625 13:51:58.654487 (+ 251us) write_op.cc:270] Start()
0625 13:51:58.654512 (+ 25us) write_op.cc:276] Timestamp: P: 1750859513104466 usec, L: 0
0625 13:51:58.654666 (+ 154us) write_op.cc:270] Start()
0625 13:51:58.654690 (+ 24us) write_op.cc:276] Timestamp: P: 1750859513118765 usec, L: 0
0625 13:51:58.654824 (+ 134us) write_op.cc:270] Start()
0625 13:51:58.654844 (+ 20us) write_op.cc:276] Timestamp: P: 1750859513154470 usec, L: 0
0625 13:51:58.654984 (+ 140us) write_op.cc:270] Start()
0625 13:51:58.655006 (+ 22us) write_op.cc:276] Timestamp: P: 1750859513174020 usec, L: 0
0625 13:51:58.655149 (+ 143us) write_op.cc:270] Start()
0625 13:51:58.655167 (+ 18us) write_op.cc:276] Timestamp: P: 1750859513192287 usec, L: 0
0625 13:51:58.655293 (+ 126us) write_op.cc:270] Start()
0625 13:51:58.655330 (+ 37us) write_op.cc:276] Timestamp: P: 1750859513210163 usec, L: 0
0625 13:51:58.655498 (+ 168us) write_op.cc:270] Start()
0625 13:51:58.655519 (+ 21us) write_op.cc:276] Timestamp: P: 1750859513215815 usec, L: 0
0625 13:51:58.655667 (+ 148us) write_op.cc:270] Start()
0625 13:51:58.655690 (+ 23us) write_op.cc:276] Timestamp: P: 1750859513222902 usec, L: 0
0625 13:51:58.655805 (+ 115us) write_op.cc:270] Start()
0625 13:51:58.655821 (+ 16us) write_op.cc:276] Timestamp: P: 1750859513240992 usec, L: 0
0625 13:51:58.656128 (+ 307us) write_op.cc:270] Start()
0625 13:51:58.656155 (+ 27us) write_op.cc:276] Timestamp: P: 1750859513252186 usec, L: 0
0625 13:51:58.656299 (+ 144us) write_op.cc:270] Start()
0625 13:51:58.656317 (+ 18us) write_op.cc:276] Timestamp: P: 1750859513255412 usec, L: 0
0625 13:51:58.656462 (+ 145us) write_op.cc:270] Start()
0625 13:51:58.656483 (+ 21us) write_op.cc:276] Timestamp: P: 1750859513374522 usec, L: 0
0625 13:51:58.656782 (+ 299us) write_op.cc:270] Start()
0625 13:51:58.656802 (+ 20us) write_op.cc:276] Timestamp: P: 1750859513389645 usec, L: 0
0625 13:51:58.656929 (+ 127us) write_op.cc:270] Start()
0625 13:51:58.656947 (+ 18us) write_op.cc:276] Timestamp: P: 1750859513401356 usec, L: 0
0625 13:51:58.657081 (+ 134us) write_op.cc:270] Start()
0625 13:51:58.657104 (+ 23us) write_op.cc:276] Timestamp: P: 1750859513413074 usec, L: 0
0625 13:51:58.657236 (+ 132us) write_op.cc:270] Start()
0625 13:51:58.657254 (+ 18us) write_op.cc:276] Timestamp: P: 1750859513428713 usec, L: 0
0625 13:51:58.657381 (+ 127us) write_op.cc:270] Start()
0625 13:51:58.657402 (+ 21us) write_op.cc:276] Timestamp: P: 1750859513446081 usec, L: 0
0625 13:51:58.657576 (+ 174us) write_op.cc:270] Start()
0625 13:51:58.657604 (+ 28us) write_op.cc:276] Timestamp: P: 1750859513461019 usec, L: 0
0625 13:51:58.657726 (+ 122us) write_op.cc:270] Start()
0625 13:51:58.657749 (+ 23us) write_op.cc:276] Timestamp: P: 1750859513480455 usec, L: 0
0625 13:51:58.657893 (+ 144us) write_op.cc:270] Start()
0625 13:51:58.657913 (+ 20us) write_op.cc:276] Timestamp: P: 1750859513494663 usec, L: 0
0625 13:51:58.658123 (+ 210us) write_op.cc:270] Start()
0625 13:51:58.658146 (+ 23us) write_op.cc:276] Timestamp: P: 1750859513500137 usec, L: 0
0625 13:51:58.658300 (+ 154us) write_op.cc:270] Start()
0625 13:51:58.658321 (+ 21us) write_op.cc:276] Timestamp: P: 1750859513501110 usec, L: 0
0625 13:51:58.658456 (+ 135us) write_op.cc:270] Start()
0625 13:51:58.658483 (+ 27us) write_op.cc:276] Timestamp: P: 1750859513502052 usec, L: 0
0625 13:51:58.658645 (+ 162us) write_op.cc:270] Start()
0625 13:51:58.658670 (+ 25us) write_op.cc:276] Timestamp: P: 1750859513516713 usec, L: 0
0625 13:51:58.658807 (+ 137us) write_op.cc:270] Start()
0625 13:51:58.658828 (+ 21us) write_op.cc:276] Timestamp: P: 1750859513520273 usec, L: 0
0625 13:51:58.658958 (+ 130us) write_op.cc:270] Start()
0625 13:51:58.658977 (+ 19us) write_op.cc:276] Timestamp: P: 1750859513537554 usec, L: 0
0625 13:51:58.659107 (+ 130us) write_op.cc:270] Start()
0625 13:51:58.659128 (+ 21us) write_op.cc:276] Timestamp: P: 1750859513548337 usec, L: 0
0625 13:51:58.659259 (+ 131us) write_op.cc:270] Start()
0625 13:51:58.659288 (+ 29us) write_op.cc:276] Timestamp: P: 1750859513552182 usec, L: 0
0625 13:51:58.659466 (+ 178us) write_op.cc:270] Start()
0625 13:51:58.659511 (+ 45us) write_op.cc:276] Timestamp: P: 1750859513558848 usec, L: 0
0625 13:51:58.659642 (+ 131us) write_op.cc:270] Start()
0625 13:51:58.659662 (+ 20us) write_op.cc:276] Timestamp: P: 1750859513562054 usec, L: 0
0625 13:51:58.659789 (+ 127us) write_op.cc:270] Start()
0625 13:51:58.659806 (+ 17us) write_op.cc:276] Timestamp: P: 1750859513562750 usec, L: 0
0625 13:51:58.659910 (+ 104us) write_op.cc:270] Start()
0625 13:51:58.659926 (+ 16us) write_op.cc:276] Timestamp: P: 1750859513563385 usec, L: 0
0625 13:51:58.660018 (+ 92us) write_op.cc:270] Start()
0625 13:51:58.674684 (+ 14666us) write_op.cc:276] Timestamp: P: 1750859513579569 usec, L: 0
0625 13:51:58.674986 (+ 302us) write_op.cc:270] Start()
0625 13:51:58.675017 (+ 31us) write_op.cc:276] Timestamp: P: 1750859513580297 usec, L: 0
0625 13:51:58.675167 (+ 150us) write_op.cc:270] Start()
0625 13:51:58.675187 (+ 20us) write_op.cc:276] Timestamp: P: 1750859513586854 usec, L: 0
0625 13:51:58.675386 (+ 199us) write_op.cc:270] Start()
0625 13:51:58.675433 (+ 47us) write_op.cc:276] Timestamp: P: 1750859513601856 usec, L: 0
0625 13:51:58.675573 (+ 140us) write_op.cc:270] Start()
0625 13:51:58.675603 (+ 30us) write_op.cc:276] Timestamp: P: 1750859513680792 usec, L: 0
0625 13:51:58.675725 (+ 122us) write_op.cc:270] Start()
0625 13:51:58.675744 (+ 19us) write_op.cc:276] Timestamp: P: 1750859513683255 usec, L: 0
0625 13:51:58.675878 (+ 134us) write_op.cc:270] Start()
0625 13:51:58.675931 (+ 53us) write_op.cc:276] Timestamp: P: 1750859513683849 usec, L: 0
0625 13:51:58.676099 (+ 168us) write_op.cc:270] Start()
0625 13:51:58.676115 (+ 16us) write_op.cc:276] Timestamp: P: 1750859513685756 usec, L: 0
0625 13:51:58.676230 (+ 115us) write_op.cc:270] Start()
0625 13:51:58.676260 (+ 30us) write_op.cc:276] Timestamp: P: 1750859513686329 usec, L: 0
0625 13:51:58.677943 (+ 1683us) write_op.cc:270] Start()
0625 13:51:58.677975 (+ 32us) write_op.cc:276] Timestamp: P: 1750859513792833 usec, L: 0
0625 13:51:58.678156 (+ 181us) write_op.cc:270] Start()
0625 13:51:58.678174 (+ 18us) write_op.cc:276] Timestamp: P: 1750859513856273 usec, L: 0
0625 13:51:58.678310 (+ 136us) write_op.cc:270] Start()
0625 13:51:58.678328 (+ 18us) write_op.cc:276] Timestamp: P: 1750859513863974 usec, L: 0
0625 13:51:58.678485 (+ 157us) write_op.cc:270] Start()
0625 13:51:58.678510 (+ 25us) write_op.cc:276] Timestamp: P: 1750859513865255 usec, L: 0
0625 13:51:58.678685 (+ 175us) write_op.cc:270] Start()
0625 13:51:58.678708 (+ 23us) write_op.cc:276] Timestamp: P: 1750859513867716 usec, L: 0
0625 13:51:58.678853 (+ 145us) write_op.cc:270] Start()
0625 13:51:58.678879 (+ 26us) write_op.cc:276] Timestamp: P: 1750859513869303 usec, L: 0
0625 13:51:58.679055 (+ 176us) write_op.cc:270] Start()
0625 13:51:58.679079 (+ 24us) write_op.cc:276] Timestamp: P: 1750859513870987 usec, L: 0
0625 13:51:58.679199 (+ 120us) write_op.cc:270] Start()
0625 13:51:58.679213 (+ 14us) write_op.cc:276] Timestamp: P: 1750859513876014 usec, L: 0
0625 13:51:58.679396 (+ 183us) write_op.cc:270] Start()
0625 13:51:58.679421 (+ 25us) write_op.cc:276] Timestamp: P: 1750859513877360 usec, L: 0
0625 13:51:58.679567 (+ 146us) write_op.cc:270] Start()
0625 13:51:58.679603 (+ 36us) write_op.cc:276] Timestamp: P: 1750859513933208 usec, L: 0
0625 13:51:58.679755 (+ 152us) write_op.cc:270] Start()
0625 13:51:58.679778 (+ 23us) write_op.cc:276] Timestamp: P: 1750859513948094 usec, L: 0
0625 13:51:58.679922 (+ 144us) write_op.cc:270] Start()
0625 13:51:58.681195 (+ 1273us) write_op.cc:276] Timestamp: P: 1750859513949357 usec, L: 0
0625 13:51:58.681563 (+ 368us) write_op.cc:270] Start()
0625 13:51:58.681585 (+ 22us) write_op.cc:276] Timestamp: P: 1750859513953220 usec, L: 0
0625 13:51:58.681731 (+ 146us) write_op.cc:270] Start()
0625 13:51:58.681748 (+ 17us) write_op.cc:276] Timestamp: P: 1750859513954380 usec, L: 0
0625 13:51:58.681857 (+ 109us) write_op.cc:270] Start()
0625 13:51:58.681872 (+ 15us) write_op.cc:276] Timestamp: P: 1750859513962167 usec, L: 0
0625 13:51:58.681967 (+ 95us) write_op.cc:270] Start()
0625 13:51:58.681987 (+ 20us) write_op.cc:276] Timestamp: P: 1750859513989251 usec, L: 0
0625 13:51:58.682099 (+ 112us) write_op.cc:270] Start()
0625 13:51:58.682114 (+ 15us) write_op.cc:276] Timestamp: P: 1750859514007903 usec, L: 0
0625 13:51:58.682239 (+ 125us) write_op.cc:270] Start()
0625 13:51:58.682254 (+ 15us) write_op.cc:276] Timestamp: P: 1750859514009662 usec, L: 0
0625 13:51:58.682361 (+ 107us) write_op.cc:270] Start()
0625 13:51:58.682377 (+ 16us) write_op.cc:276] Timestamp: P: 1750859514010785 usec, L: 0
0625 13:51:58.682515 (+ 138us) write_op.cc:270] Start()
0625 13:51:58.682540 (+ 25us) write_op.cc:276] Timestamp: P: 1750859514012659 usec, L: 0
0625 13:51:58.682705 (+ 165us) write_op.cc:270] Start()
0625 13:51:58.682728 (+ 23us) write_op.cc:276] Timestamp: P: 1750859514028311 usec, L: 0
0625 13:51:58.682904 (+ 176us) write_op.cc:270] Start()
0625 13:51:58.682928 (+ 24us) write_op.cc:276] Timestamp: P: 1750859514068438 usec, L: 0
0625 13:51:58.683084 (+ 156us) write_op.cc:270] Start()
0625 13:51:58.683107 (+ 23us) write_op.cc:276] Timestamp: P: 1750859514099471 usec, L: 0
0625 13:51:58.687732 (+ 4625us) write_op.cc:270] Start()
0625 13:51:58.687768 (+ 36us) write_op.cc:276] Timestamp: P: 1750859514100956 usec, L: 0
0625 13:51:58.688185 (+ 417us) write_op.cc:270] Start()
0625 13:51:58.688213 (+ 28us) write_op.cc:276] Timestamp: P: 1750859514102400 usec, L: 0
0625 13:51:58.688352 (+ 139us) write_op.cc:270] Start()
0625 13:51:58.688370 (+ 18us) write_op.cc:
W20250625 13:51:59.434252 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ed3c177bb6dd488aa54bee2a447016ff (table default.Table_c4f1a08649504a7db750c88b65ff985c [id=4631cf687a9c478ba42e9ea3375ca639], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ed3c177bb6dd488aa54bee2a447016ff consensus state has no leader
I20250625 13:51:59.441479 1455 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "affa1c3035b940e88ca143be7d35840d" instance_seqno: 1750859510608079) as {username='slave'} at 127.0.104.2:49165; Asking this server to re-register.
I20250625 13:51:59.442133 1521 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:51:59.442364 1521 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: LEADER
I20250625 13:51:59.444023 961 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:59.447767 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc9536a716f4b9f9fc4af1637b3cd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.449717 1170 tablet_bootstrap.cc:654] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.451637 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35aedb3b590d4917bac8fe0962d68fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.455621 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb648c45ea344fb3bac765c9e06db2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.459645 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.460270 1170 tablet_bootstrap.cc:492] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:59.460556 1170 ts_tablet_manager.cc:1397] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.004s
I20250625 13:51:59.457363 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f5780d88dea41e7b855b41f12e96c52 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.463233 1170 raft_consensus.cc:357] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.463550 1170 raft_consensus.cc:383] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.471464 1373 raft_consensus.cc:491] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.471411 891 tablet_service.cc:1468] Processing CreateTablet for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 (DEFAULT_TABLE table=default.Table_0e7f3b83e2c7403eb46bc273da03cee8 [id=1286a09e25024ae58684376e9919eb62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.472014 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb648c45ea344fb3bac765c9e06db2b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.472249 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35aedb3b590d4917bac8fe0962d68fb3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.474035 711 leader_election.cc:343] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:51:59.474273 711 leader_election.cc:343] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.474392 1373 raft_consensus.cc:513] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.475104 711 leader_election.cc:304] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.476071 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "583ee2f2c66841939c93104efc36facb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:59.476313 913 raft_consensus.cc:2466] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:51:59.476845 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.477288 1165 ts_tablet_manager.cc:1918] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.478313 1165 log.cc:1199] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/9ba63184ca2b4badafcf08309ef75164
I20250625 13:51:59.478991 1165 ts_tablet_manager.cc:1939] T 9ba63184ca2b4badafcf08309ef75164 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.480098 1607 raft_consensus.cc:2747] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:59.480191 711 leader_election.cc:304] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:59.480921 1373 leader_election.cc:290] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:59.481531 1168 ts_tablet_manager.cc:1918] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.481675 1168 log.cc:1199] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/67441795e22b445a90c88d76aa519ea5
I20250625 13:51:59.482470 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5d5a62a1608c45929c8fa1c853a06a27" } modify_external_catalogs: true
I20250625 13:51:59.482527 886 tablet_service.cc:1468] Processing CreateTablet for tablet e73bea55af9e4890bf7070b488313c3f (DEFAULT_TABLE table=default.Table_528b80e737c9462194cd9b50d91b7db9 [id=171d9555defd4d1ba8634fb0edfb55d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.482658 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5d5a62a1608c45929c8fa1c853a06a27" } modify_external_catalogs: true
I20250625 13:51:59.482949 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73bea55af9e4890bf7070b488313c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.483537 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e78b8ab5d2584ab19f2560a2298b21cf" } new_table_name: "default.Table_cb91eec0b55a4cd1b261ee9f113a4644" modify_external_catalogs: true
I20250625 13:51:59.484874 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0892d2b459334a8bac680a69f2be033e" } new_table_name: "default.Table_cdcf80766e3e4d9c98920cc3d711764d" modify_external_catalogs: true
I20250625 13:51:59.485302 883 tablet_service.cc:1468] Processing CreateTablet for tablet efc9536a716f4b9f9fc4af1637b3cd80 (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.485656 1510 raft_consensus.cc:491] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.487387 1373 raft_consensus.cc:2802] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:59.487511 1373 raft_consensus.cc:491] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:59.487854 1373 raft_consensus.cc:3058] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:59.488139 849 leader_election.cc:343] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:51:59.488269 849 leader_election.cc:304] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:51:59.488922 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 44277.9us
W20250625 13:51:59.489331 849 leader_election.cc:343] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 7108932fa8a241819ca90674a64b285e
I20250625 13:51:59.490913 849 leader_election.cc:304] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:59.491101 748 tablet_service.cc:1468] Processing CreateTablet for tablet efc9536a716f4b9f9fc4af1637b3cd80 (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.491533 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc9536a716f4b9f9fc4af1637b3cd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.491551 1442 raft_consensus.cc:2747] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:51:59.491746 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc9536a716f4b9f9fc4af1637b3cd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.492172 1510 raft_consensus.cc:513] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.492738 1510 leader_election.cc:290] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:59.494630 1512 raft_consensus.cc:2747] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:51:59.496742 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6be7189f8f45ccb329ef5d2fd8da8d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:59.497366 711 leader_election.cc:343] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.500689 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ba63184ca2b4badafcf08309ef75164" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:59.501456 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6be7189f8f45ccb329ef5d2fd8da8d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:59.501941 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 356) took 1975 ms. Trace:
W20250625 13:51:59.502018 711 leader_election.cc:343] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.502180 711 leader_election.cc:304] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.502635 1607 raft_consensus.cc:2747] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:59.502102 1165 rpcz_store.cc:276] 0625 13:51:57.504884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.504996 (+ 112us) service_pool.cc:225] Handling call
0625 13:51:59.369032 (+1864036us) ts_tablet_manager.cc:1145] Deleting tablet 9ba63184ca2b4badafcf08309ef75164
0625 13:51:59.369041 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.379764 (+ 10723us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.477170 (+ 97406us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.479988 (+ 2818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863858}
I20250625 13:51:59.503139 1373 raft_consensus.cc:513] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.503654 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 357) took 1970 ms. Trace:
I20250625 13:51:59.503787 1165 rpcz_store.cc:276] 0625 13:51:57.532973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.533094 (+ 121us) service_pool.cc:225] Handling call
0625 13:51:59.503450 (+1970356us) ts_tablet_manager.cc:1145] Deleting tablet a0e81d4c904248eba858462bcf31c5d9
0625 13:51:59.503460 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.503581 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1952970}
I20250625 13:51:59.503836 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 9ba63184ca2b4badafcf08309ef75164 (table default.Table_dbca3ace8835418e8db2cbc1845d9cfb [id=1cd7ba4de230481e8b0c57c1c585fbd2]) successfully deleted
I20250625 13:51:59.504112 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 430c3bed360b4f32855a8adb99076608 with delete_type TABLET_DATA_DELETED (replaced by tablet 0f63e9037bd74f1eb39bd85aa820ec35) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.508143 1455 ts_manager.cc:194] Registered new tserver with Master: affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:59.509079 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "583ee2f2c66841939c93104efc36facb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:59.504794 1373 leader_election.cc:290] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:59.510602 711 leader_election.cc:343] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.504357 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 360) took 1893 ms. Trace:
I20250625 13:51:59.511121 1165 rpcz_store.cc:276] 0625 13:51:57.611332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.612077 (+ 745us) service_pool.cc:225] Handling call
0625 13:51:59.504212 (+1892135us) ts_tablet_manager.cc:1145] Deleting tablet 1effed0fa10d409abcb31dbba54861d6
0625 13:51:59.504220 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.504301 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891927}
W20250625 13:51:59.511582 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a0e81d4c904248eba858462bcf31c5d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e81d4c904248eba858462bcf31c5d9 already in progress: creating tablet
W20250625 13:51:59.511817 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1effed0fa10d409abcb31dbba54861d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1effed0fa10d409abcb31dbba54861d6 already in progress: creating tablet
I20250625 13:51:59.512089 1165 tablet_replica.cc:331] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:59.512264 1165 raft_consensus.cc:2241] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.512434 1165 raft_consensus.cc:2270] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.512885 1165 ts_tablet_manager.cc:1905] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.515555 1168 ts_tablet_manager.cc:1939] T 67441795e22b445a90c88d76aa519ea5 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:59.515784 1171 tablet_bootstrap.cc:654] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.517158 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 180) took 4808 ms. Trace:
W20250625 13:51:59.518765 851 leader_election.cc:343] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9ba63184ca2b4badafcf08309ef75164
I20250625 13:51:59.519579 1165 ts_tablet_manager.cc:1918] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.519783 1165 log.cc:1199] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/02f2a2b595bc43309a29b3f1bc5d3766
I20250625 13:51:59.520241 1165 ts_tablet_manager.cc:1939] T 02f2a2b595bc43309a29b3f1bc5d3766 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.520861 1168 rpcz_store.cc:276] 0625 13:51:54.708367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.711819 (+ 3452us) service_pool.cc:225] Handling call
0625 13:51:59.163508 (+4451689us) ts_tablet_manager.cc:1145] Deleting tablet 67441795e22b445a90c88d76aa519ea5
0625 13:51:59.163521 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.197046 (+ 33525us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.406349 (+209303us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.517095 (+110746us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4451413}
I20250625 13:51:59.521358 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 189) took 4691 ms. Trace:
I20250625 13:51:59.521379 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 361) took 1909 ms. Trace:
I20250625 13:51:59.521466 1168 rpcz_store.cc:276] 0625 13:51:54.830367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.830423 (+ 56us) service_pool.cc:225] Handling call
0625 13:51:59.521205 (+4690782us) ts_tablet_manager.cc:1145] Deleting tablet d0c5bea3b6214fa8af2792c5e9c411d9
0625 13:51:59.521216 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.521329 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4689525}
I20250625 13:51:59.521485 1165 rpcz_store.cc:276] 0625 13:51:57.611654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.612316 (+ 662us) service_pool.cc:225] Handling call
0625 13:51:59.512019 (+1899703us) ts_tablet_manager.cc:1145] Deleting tablet 02f2a2b595bc43309a29b3f1bc5d3766
0625 13:51:59.512027 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.515116 (+ 3089us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.519416 (+ 4300us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.521340 (+ 1924us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898636}
I20250625 13:51:59.521838 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 193) took 4542 ms. Trace:
I20250625 13:51:59.521960 1168 rpcz_store.cc:276] 0625 13:51:54.981528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.982319 (+ 791us) service_pool.cc:225] Handling call
0625 13:51:59.521691 (+4539372us) ts_tablet_manager.cc:1145] Deleting tablet 2cd5303e674246ada7d36574b535b333
0625 13:51:59.521711 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.521798 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4539124}
I20250625 13:51:59.522038 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 02f2a2b595bc43309a29b3f1bc5d3766 (table default.Table_aacf50f903c54d17a68e13722c931d22 [id=8c054766f5424144a4edb39b192451cb]) successfully deleted
I20250625 13:51:59.522266 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 188) took 4694 ms. Trace:
I20250625 13:51:59.522384 1168 rpcz_store.cc:276] 0625 13:51:54.828243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:54.876388 (+ 48145us) service_pool.cc:225] Handling call
0625 13:51:59.522165 (+4645777us) ts_tablet_manager.cc:1145] Deleting tablet 15e58fbec94e4ff9b7ac9830b576fe7f
0625 13:51:59.522172 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.522238 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4531815}
I20250625 13:51:59.523241 1168 tablet_replica.cc:331] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:59.523487 1168 raft_consensus.cc:2241] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.523658 1168 raft_consensus.cc:2270] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.524194 1168 ts_tablet_manager.cc:1905] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.524720 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 373) took 1700 ms. Trace:
I20250625 13:51:59.524829 1165 rpcz_store.cc:276] 0625 13:51:57.824058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.824721 (+ 663us) service_pool.cc:225] Handling call
0625 13:51:59.524578 (+1699857us) ts_tablet_manager.cc:1145] Deleting tablet 32a57268a18f4f00a39f8360f9c90737
0625 13:51:59.524589 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.524688 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698916}
W20250625 13:51:59.525341 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 32a57268a18f4f00a39f8360f9c90737 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a57268a18f4f00a39f8360f9c90737 already in progress: creating tablet
I20250625 13:51:59.525946 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 374) took 1701 ms. Trace:
I20250625 13:51:59.526136 1165 rpcz_store.cc:276] 0625 13:51:57.824151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.824744 (+ 593us) service_pool.cc:225] Handling call
0625 13:51:59.525728 (+1700984us) ts_tablet_manager.cc:1145] Deleting tablet 37e4bd4aaab143ffa65a668a952ae345
0625 13:51:59.525743 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.525899 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1696895}
W20250625 13:51:59.526819 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 37e4bd4aaab143ffa65a668a952ae345 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37e4bd4aaab143ffa65a668a952ae345 already in progress: creating tablet
I20250625 13:51:59.527441 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "583ee2f2c66841939c93104efc36facb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:51:59.527658 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 372) took 1703 ms. Trace:
I20250625 13:51:59.527769 913 raft_consensus.cc:3058] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.527819 1165 rpcz_store.cc:276] 0625 13:51:57.823955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.824701 (+ 746us) service_pool.cc:225] Handling call
0625 13:51:59.527521 (+1702820us) ts_tablet_manager.cc:1145] Deleting tablet cd4f0857f52449e1b26fde09c6c7c9d4
0625 13:51:59.527531 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.527623 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698367}
W20250625 13:51:59.528357 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet cd4f0857f52449e1b26fde09c6c7c9d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd4f0857f52449e1b26fde09c6c7c9d4 already in progress: creating tablet
W20250625 13:51:59.528668 849 leader_election.cc:343] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.528746 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 375) took 1692 ms. Trace:
I20250625 13:51:59.528939 1165 rpcz_store.cc:276] 0625 13:51:57.836212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.836285 (+ 73us) service_pool.cc:225] Handling call
0625 13:51:59.528631 (+1692346us) ts_tablet_manager.cc:1145] Deleting tablet 68b8c8aa8036454298bf1e81bab91c01
0625 13:51:59.528639 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.528717 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664302}
W20250625 13:51:59.529336 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 68b8c8aa8036454298bf1e81bab91c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68b8c8aa8036454298bf1e81bab91c01 already in progress: creating tablet
I20250625 13:51:59.529825 1165 tablet_replica.cc:331] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:59.530087 1165 raft_consensus.cc:2241] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.530398 1165 raft_consensus.cc:2270] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.537201 1170 raft_consensus.cc:738] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.537606 1170 consensus_queue.cc:260] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.538465 1170 ts_tablet_manager.cc:1428] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.078s user 0.002s sys 0.000s
I20250625 13:51:59.538872 1170 tablet_bootstrap.cc:492] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:59.540580 913 raft_consensus.cc:2466] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:51:59.541546 711 leader_election.cc:304] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:59.541620 849 leader_election.cc:304] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:59.541877 1373 raft_consensus.cc:2802] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:59.542089 1373 raft_consensus.cc:695] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:51:59.542104 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_478b0037c80e40048d4f8ebc92748f34" } modify_external_catalogs: true
I20250625 13:51:59.542304 1512 raft_consensus.cc:2747] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:59.542326 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_478b0037c80e40048d4f8ebc92748f34" } modify_external_catalogs: true
I20250625 13:51:59.542375 1373 consensus_queue.cc:237] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.542945 1165 ts_tablet_manager.cc:1905] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.550097 1049 raft_consensus.cc:3058] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:59.550415 849 consensus_peers.cc:487] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 412a0c5f05ab489ab9f8fc3f44fa7bdc. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:59.554294 1512 consensus_queue.cc:1035] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:59.556699 1512 consensus_queue.cc:1035] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:59.555733 1167 ts_tablet_manager.cc:1918] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.559269 1167 log.cc:1199] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/7fe053e294fe4aceb24919e9e2bddf65
I20250625 13:51:59.559768 1167 ts_tablet_manager.cc:1939] T 7fe053e294fe4aceb24919e9e2bddf65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:51:59.560086 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 304) took 2503 ms. Trace:
I20250625 13:51:59.560218 1167 rpcz_store.cc:276] 0625 13:51:57.056829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.056953 (+ 124us) service_pool.cc:225] Handling call
0625 13:51:59.223764 (+2166811us) ts_tablet_manager.cc:1145] Deleting tablet 7fe053e294fe4aceb24919e9e2bddf65
0625 13:51:59.223775 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.513498 (+289723us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.555569 (+ 42071us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.560057 (+ 4488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2163519}
I20250625 13:51:59.562399 1170 tablet_bootstrap.cc:654] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.566139 1168 ts_tablet_manager.cc:1918] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.566278 1166 tablet_bootstrap.cc:492] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.566457 1166 ts_tablet_manager.cc:1397] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.238s user 0.012s sys 0.045s
I20250625 13:51:59.567534 1166 raft_consensus.cc:357] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.567787 1166 raft_consensus.cc:383] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.571623 1166 raft_consensus.cc:738] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.571916 1166 consensus_queue.cc:260] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.571012 1512 raft_consensus.cc:491] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.573129 1512 raft_consensus.cc:513] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:59.573915 1166 ts_tablet_manager.cc:1428] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
W20250625 13:51:59.575167 849 leader_election.cc:343] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 32221a413957450fa59ffcfbc91c51b9
I20250625 13:51:59.574400 1512 leader_election.cc:290] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:59.575919 1168 log.cc:1199] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/14a12a5453e647b9946ca19e0ffb14ee
W20250625 13:51:59.576342 851 leader_election.cc:343] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 32221a413957450fa59ffcfbc91c51b9
I20250625 13:51:59.576449 1168 ts_tablet_manager.cc:1939] T 14a12a5453e647b9946ca19e0ffb14ee P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:59.576510 851 leader_election.cc:304] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:59.576761 1512 raft_consensus.cc:2747] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 13:51:59.579161 581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:33015, user_credentials={real_user=slave}} blocked reactor thread for 68027.2us
I20250625 13:51:59.579578 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 67441795e22b445a90c88d76aa519ea5 (table default.Table_5d5a62a1608c45929c8fa1c853a06a27 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]) successfully deleted
W20250625 13:51:59.579864 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d0c5bea3b6214fa8af2792c5e9c411d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0c5bea3b6214fa8af2792c5e9c411d9 already in progress: creating tablet
W20250625 13:51:59.580080 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2cd5303e674246ada7d36574b535b333 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cd5303e674246ada7d36574b535b333 already in progress: creating tablet
W20250625 13:51:59.580279 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 15e58fbec94e4ff9b7ac9830b576fe7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15e58fbec94e4ff9b7ac9830b576fe7f already in progress: creating tablet
I20250625 13:51:59.580935 1166 tablet_bootstrap.cc:492] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:51:59.581687 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cf1664afbab647b3baeed6391196bed9
I20250625 13:51:59.582512 757 tablet_service.cc:1515] Processing DeleteTablet for tablet cf1664afbab647b3baeed6391196bed9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.582974 1171 tablet_bootstrap.cc:492] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:59.583150 1171 ts_tablet_manager.cc:1397] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.356s user 0.006s sys 0.001s
I20250625 13:51:59.584066 1171 raft_consensus.cc:357] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.584314 1171 raft_consensus.cc:383] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.584391 1171 raft_consensus.cc:738] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.584825 1171 consensus_queue.cc:260] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.585724 1165 ts_tablet_manager.cc:1918] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:59.585861 1165 log.cc:1199] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d524089e6ab044ec961cfc9f587e8f86
I20250625 13:51:59.586259 1165 ts_tablet_manager.cc:1939] T d524089e6ab044ec961cfc9f587e8f86 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.587121 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 376) took 1725 ms. Trace:
I20250625 13:51:59.587395 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 413afd7b62d24fe8b9eceadd57e29fc8
I20250625 13:51:59.588239 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.588603 1165 rpcz_store.cc:276] 0625 13:51:57.861725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.861956 (+ 231us) service_pool.cc:225] Handling call
0625 13:51:59.529761 (+1667805us) ts_tablet_manager.cc:1145] Deleting tablet d524089e6ab044ec961cfc9f587e8f86
0625 13:51:59.529769 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.548697 (+ 18928us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.585601 (+ 36904us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.587066 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648221}
I20250625 13:51:59.589121 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d524089e6ab044ec961cfc9f587e8f86 (table default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]) successfully deleted
I20250625 13:51:59.589748 1166 tablet_bootstrap.cc:654] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.589915 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_af5d40e690b54d16b4689c880fc20261"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.590507 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af5d40e690b54d16b4689c880fc20261 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.590893 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b5fe7dfea5c84fc4a7cc420c24d1e54b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.591419 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5fe7dfea5c84fc4a7cc420c24d1e54b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.591742 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_25010c89df9e4e128b8e4531a0bda6d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.591959 757 tablet_service.cc:1515] Processing DeleteTablet for tablet c911d9ecbded417daf70e85c7cbcd52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:51:59.592170 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25010c89df9e4e128b8e4531a0bda6d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.593665 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aacf50f903c54d17a68e13722c931d22" } new_table_name: "default.Table_630c47cb56984efbac74836e412efaa6" modify_external_catalogs: true
I20250625 13:51:59.596609 1171 ts_tablet_manager.cc:1428] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250625 13:51:59.598191 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 7fe053e294fe4aceb24919e9e2bddf65 (table default.Table_d9dd2f12f8bb4f6cba435c892dac832e [id=98f8967275434f48a948def71ae1d693]) successfully deleted
I20250625 13:51:59.599083 886 tablet_service.cc:1515] Processing DeleteTablet for tablet cf1664afbab647b3baeed6391196bed9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.599833 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bdcf17e5612a44579ba27af546ca1def" } modify_external_catalogs: true
I20250625 13:51:59.600324 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bdcf17e5612a44579ba27af546ca1def" } modify_external_catalogs: true
I20250625 13:51:59.602681 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 380) took 1529 ms. Trace:
I20250625 13:51:59.603596 1165 rpcz_store.cc:276] 0625 13:51:58.073609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.074503 (+ 894us) service_pool.cc:225] Handling call
0625 13:51:59.602412 (+1527909us) ts_tablet_manager.cc:1145] Deleting tablet db6e9db2375144b38ec43ffca9ca4c1e
0625 13:51:59.602424 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.602566 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502804}
W20250625 13:51:59.605346 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet db6e9db2375144b38ec43ffca9ca4c1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db6e9db2375144b38ec43ffca9ca4c1e already in progress: creating tablet
I20250625 13:51:59.608140 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_63e7af63093f48dd8b6e416d8676fade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.608762 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63e7af63093f48dd8b6e416d8676fade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.610415 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7bd4939714a4359846345ed0521701c consensus state has no leader
W20250625 13:51:59.612752 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 024f6cb363594420b0d5480fa79d156b consensus state has no leader
I20250625 13:51:59.611158 1171 tablet_bootstrap.cc:492] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:59.613312 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet abd6f9121a9d4b39adde1b880a875e23
I20250625 13:51:59.614634 757 tablet_service.cc:1515] Processing DeleteTablet for tablet abd6f9121a9d4b39adde1b880a875e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.614681 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.614933 1171 tablet_bootstrap.cc:654] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.615411 886 tablet_service.cc:1515] Processing DeleteTablet for tablet abd6f9121a9d4b39adde1b880a875e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.616191 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 41a7862601f04fd7a938f7e5d5a4c0f9
I20250625 13:51:59.616887 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0220bc5deb894b909ecd95a1724cf43d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.616976 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_81732e5a747642cd8d61556c2ed6f6c7" } new_table_name: "default.Table_ff092aaea4a944aba8072d10f3b7ce49" modify_external_catalogs: true
W20250625 13:51:59.617403 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0220bc5deb894b909ecd95a1724cf43d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.619433 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.619916 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.623035 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_06ce856b6c374443991c4f9ce9a60eb1" } new_table_name: "default.Table_ac4d598c675a46d1bbd2a08d9652f2c7" modify_external_catalogs: true
I20250625 13:51:59.624583 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_865a0d33634943c88b8a9b87a6cae4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.625160 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_865a0d33634943c88b8a9b87a6cae4b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.612872 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 194) took 4597 ms. Trace:
I20250625 13:51:59.626251 1168 rpcz_store.cc:276] 0625 13:51:55.014513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.014582 (+ 69us) service_pool.cc:225] Handling call
0625 13:51:59.522554 (+4507972us) ts_tablet_manager.cc:1145] Deleting tablet 14a12a5453e647b9946ca19e0ffb14ee
0625 13:51:59.522562 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.562513 (+ 39951us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.566030 (+ 3517us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.611589 (+ 45559us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x61800000d518. stack: 00007f37024dc336 00007f37024dc0f0 00007f37014d23ad 0000000000971004 0000000000970fcd 000000000097099a 00007f37024c4c3d 00007f37024c5cae 00007f3715b5a80d 00007f3715adc0f3 00007f3711d2b5a3 00007f3711d5d1c4 00007f3711d3ec20 00007f3711d3e9ad 0000000000a7573c 00007f37025219df
0625 13:51:59.612212 (+ 623us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5195520,"tablet-delete.queue_time_us":4505102}
I20250625 13:51:59.626716 1168 tablet_replica.cc:331] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:59.626945 1168 raft_consensus.cc:2241] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:51:59.627184 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3939ef22ea344328721dfba67d933d1 consensus state has no leader
I20250625 13:51:59.629467 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 14a12a5453e647b9946ca19e0ffb14ee (table default.Table_6cfe68ee62114f35be36b7f12d451a24 [id=09a0b05c11214fd0952325f08a002b8b]) successfully deleted
I20250625 13:51:59.633858 1512 raft_consensus.cc:491] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.634045 1512 raft_consensus.cc:513] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.634910 1512 leader_election.cc:290] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:59.634061 1168 raft_consensus.cc:2270] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.636039 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50786a4c25274ffb8f9910c5dafe3933" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:59.636292 777 raft_consensus.cc:2466] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:51:59.637079 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50786a4c25274ffb8f9910c5dafe3933" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:51:59.637748 849 leader_election.cc:343] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.638124 851 leader_election.cc:304] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:51:59.638845 1510 raft_consensus.cc:2802] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:51:59.638945 1510 raft_consensus.cc:491] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:51:59.639045 1510 raft_consensus.cc:3058] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.640079 1168 ts_tablet_manager.cc:1905] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.641842 1518 raft_consensus.cc:491] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.642170 1510 raft_consensus.cc:513] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.645444 1510 leader_election.cc:290] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:59.645411 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50786a4c25274ffb8f9910c5dafe3933" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:51:59.645668 776 raft_consensus.cc:3058] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:51:59.645913 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e2bc4df39674dc3be0cd32c68f1ebca consensus state has no leader
W20250625 13:51:59.646598 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 423ca098138c476f861a19b44554fd24 consensus state has no leader
I20250625 13:51:59.650915 776 raft_consensus.cc:2466] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:51:59.651561 851 leader_election.cc:304] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:51:59.652566 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 300) took 2531 ms. Trace:
I20250625 13:51:59.653067 1510 raft_consensus.cc:2802] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:51:59.655213 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.655539 1518 raft_consensus.cc:513] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.654765 1510 raft_consensus.cc:695] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:51:59.656803 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 77c50e1ebb9042ef8f0a651ba7186b74
I20250625 13:51:59.656868 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:51:59.657706 1171 tablet_bootstrap.cc:492] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:59.657864 1171 ts_tablet_manager.cc:1397] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.003s
I20250625 13:51:59.657997 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.658133 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50786a4c25274ffb8f9910c5dafe3933" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:51:59.657305 1510 consensus_queue.cc:237] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.658699 1171 raft_consensus.cc:357] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.658891 1171 raft_consensus.cc:383] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.658936 1170 tablet_bootstrap.cc:492] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:59.658993 1171 raft_consensus.cc:738] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.659090 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet abd6f9121a9d4b39adde1b880a875e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.659116 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.659209 1171 consensus_queue.cc:260] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:51:59.659530 849 leader_election.cc:343] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.659825 1171 ts_tablet_manager.cc:1428] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:51:59.660107 1171 tablet_bootstrap.cc:492] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:59.660430 1170 ts_tablet_manager.cc:1397] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.120s user 0.005s sys 0.012s
I20250625 13:51:59.661821 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4f19ddc51324cc18242c6081f3f7de5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:59.661955 1170 raft_consensus.cc:357] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.662375 1171 tablet_bootstrap.cc:654] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.663353 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.664187 1167 rpcz_store.cc:276] 0625 13:51:57.049194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.049603 (+ 409us) service_pool.cc:225] Handling call
0625 13:51:59.570775 (+2521172us) ts_tablet_manager.cc:1145] Deleting tablet 29be8c0350584de382985e66556bdcca
0625 13:51:59.570785 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.580556 (+ 9771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2510149}
I20250625 13:51:59.664619 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 301) took 2615 ms. Trace:
I20250625 13:51:59.664737 1167 rpcz_store.cc:276] 0625 13:51:57.049370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.049631 (+ 261us) service_pool.cc:225] Handling call
0625 13:51:59.664455 (+2614824us) ts_tablet_manager.cc:1145] Deleting tablet 7ba1338d0562418eadabdb8dbcaedee2
0625 13:51:59.664464 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.664588 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2603780}
I20250625 13:51:59.665060 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 328) took 2286 ms. Trace:
I20250625 13:51:59.665162 1167 rpcz_store.cc:276] 0625 13:51:57.378310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.481783 (+103473us) service_pool.cc:225] Handling call
0625 13:51:59.664947 (+2183164us) ts_tablet_manager.cc:1145] Deleting tablet 5d7704b38144429b9c95675be84a4730
0625 13:51:59.664956 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.665031 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2109112}
I20250625 13:51:59.665483 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 331) took 2286 ms. Trace:
I20250625 13:51:59.665591 1167 rpcz_store.cc:276] 0625 13:51:57.378619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.558486 (+179867us) service_pool.cc:225] Handling call
0625 13:51:59.665362 (+2106876us) ts_tablet_manager.cc:1145] Deleting tablet f73f420a4d7a438eadde3d6af64f0a11
0625 13:51:59.665371 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.665456 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2106651}
I20250625 13:51:59.665912 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 321) took 2302 ms. Trace:
I20250625 13:51:59.666002 1167 rpcz_store.cc:276] 0625 13:51:57.363557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.479757 (+116200us) service_pool.cc:225] Handling call
0625 13:51:59.665799 (+2186042us) ts_tablet_manager.cc:1145] Deleting tablet d9bb7ebc40a14ecaa0c692a8e1117db3
0625 13:51:59.665814 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.665881 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2106877}
I20250625 13:51:59.666271 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 333) took 2287 ms. Trace:
I20250625 13:51:59.666363 1167 rpcz_store.cc:276] 0625 13:51:57.378847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.558937 (+180090us) service_pool.cc:225] Handling call
0625 13:51:59.666172 (+2107235us) ts_tablet_manager.cc:1145] Deleting tablet 87f5b36ddb474ed6903c14876b236ba8
0625 13:51:59.666178 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.666243 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2107086}
I20250625 13:51:59.666633 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 322) took 2292 ms. Trace:
I20250625 13:51:59.666721 1167 rpcz_store.cc:276] 0625 13:51:57.374614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.479945 (+105331us) service_pool.cc:225] Handling call
0625 13:51:59.666527 (+2186582us) ts_tablet_manager.cc:1145] Deleting tablet 9514bbbda0ba4329a18b0a42e635c6e5
0625 13:51:59.666534 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.666597 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2101641}
I20250625 13:51:59.667011 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 329) took 2288 ms. Trace:
I20250625 13:51:59.667110 1167 rpcz_store.cc:276] 0625 13:51:57.378397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.507807 (+129410us) service_pool.cc:225] Handling call
0625 13:51:59.666897 (+2159090us) ts_tablet_manager.cc:1145] Deleting tablet 63d01af6b320489fb99c467c11ed1dc0
0625 13:51:59.666904 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.666984 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096189}
W20250625 13:51:59.667972 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 29be8c0350584de382985e66556bdcca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29be8c0350584de382985e66556bdcca already in progress: creating tablet
W20250625 13:51:59.668234 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7ba1338d0562418eadabdb8dbcaedee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba1338d0562418eadabdb8dbcaedee2 already in progress: creating tablet
W20250625 13:51:59.668402 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5d7704b38144429b9c95675be84a4730 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d7704b38144429b9c95675be84a4730 already in progress: creating tablet
W20250625 13:51:59.668581 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f73f420a4d7a438eadde3d6af64f0a11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f73f420a4d7a438eadde3d6af64f0a11 already in progress: creating tablet
W20250625 13:51:59.668756 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9bb7ebc40a14ecaa0c692a8e1117db3 already in progress: creating tablet
W20250625 13:51:59.669015 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9514bbbda0ba4329a18b0a42e635c6e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9514bbbda0ba4329a18b0a42e635c6e5 already in progress: creating tablet
W20250625 13:51:59.669194 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 63d01af6b320489fb99c467c11ed1dc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63d01af6b320489fb99c467c11ed1dc0 already in progress: creating tablet
I20250625 13:51:59.669440 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.669596 1518 leader_election.cc:290] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:59.670531 983 leader_election.cc:304] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.670636 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 330) took 2292 ms. Trace:
I20250625 13:51:59.670802 1167 rpcz_store.cc:276] 0625 13:51:57.378520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.555872 (+177352us) service_pool.cc:225] Handling call
0625 13:51:59.667292 (+2111420us) ts_tablet_manager.cc:1145] Deleting tablet 4fa801f7d98e4794b6ca439638f29e69
0625 13:51:59.667299 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.670598 (+ 3299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096068}
I20250625 13:51:59.671298 1170 raft_consensus.cc:383] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.671376 1518 raft_consensus.cc:2747] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:51:59.671432 1170 raft_consensus.cc:738] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:51:59.672142 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4fa801f7d98e4794b6ca439638f29e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fa801f7d98e4794b6ca439638f29e69 already in progress: creating tablet
I20250625 13:51:59.672477 1170 consensus_queue.cc:260] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.673038 1170 ts_tablet_manager.cc:1428] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250625 13:51:59.673528 1170 tablet_bootstrap.cc:492] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:59.674839 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 332) took 2295 ms. Trace:
I20250625 13:51:59.676718 1167 rpcz_store.cc:276] 0625 13:51:57.378741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.558734 (+179993us) service_pool.cc:225] Handling call
0625 13:51:59.671047 (+2112313us) ts_tablet_manager.cc:1145] Deleting tablet 70959ec681524518a7ec2c9ed4016ad1
0625 13:51:59.671057 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.674690 (+ 3633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2099120}
I20250625 13:51:59.678328 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 390) took 1349 ms. Trace:
I20250625 13:51:59.678465 1165 rpcz_store.cc:276] 0625 13:51:58.328538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.328908 (+ 370us) service_pool.cc:225] Handling call
0625 13:51:59.678084 (+1349176us) ts_tablet_manager.cc:1145] Deleting tablet 535812d46d6444fda85d9767c6ca7e62
0625 13:51:59.678101 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.678252 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348959}
I20250625 13:51:59.679348 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 345) took 2143 ms. Trace:
W20250625 13:51:59.679898 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 70959ec681524518a7ec2c9ed4016ad1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70959ec681524518a7ec2c9ed4016ad1 already in progress: creating tablet
I20250625 13:51:59.686228 1167 rpcz_store.cc:276] 0625 13:51:57.536169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.792235 (+256066us) service_pool.cc:225] Handling call
0625 13:51:59.679145 (+1886910us) ts_tablet_manager.cc:1145] Deleting tablet ac4d368347444846880954ced9651404
0625 13:51:59.679159 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.679279 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882118}
W20250625 13:51:59.687423 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ac4d368347444846880954ced9651404 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac4d368347444846880954ced9651404 already in progress: creating tablet
I20250625 13:51:59.687983 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 346) took 2151 ms. Trace:
I20250625 13:51:59.688587 1167 rpcz_store.cc:276] 0625 13:51:57.536272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.798137 (+261865us) service_pool.cc:225] Handling call
0625 13:51:59.687790 (+1889653us) ts_tablet_manager.cc:1145] Deleting tablet a22ade8cefac4140b08882c311ea5181
0625 13:51:59.687811 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.687942 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1881420}
W20250625 13:51:59.689318 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a22ade8cefac4140b08882c311ea5181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a22ade8cefac4140b08882c311ea5181 already in progress: creating tablet
I20250625 13:51:59.688130 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet cf1664afbab647b3baeed6391196bed9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.684114 1170 tablet_bootstrap.cc:654] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.690955 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 348) took 2154 ms. Trace:
I20250625 13:51:59.691365 1167 rpcz_store.cc:276] 0625 13:51:57.536489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.805889 (+269400us) service_pool.cc:225] Handling call
0625 13:51:59.690763 (+1884874us) ts_tablet_manager.cc:1145] Deleting tablet 9ba63184ca2b4badafcf08309ef75164
0625 13:51:59.690773 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.690910 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879595}
W20250625 13:51:59.695677 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9ba63184ca2b4badafcf08309ef75164 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ba63184ca2b4badafcf08309ef75164 already in progress: creating tablet
I20250625 13:51:59.696107 1168 ts_tablet_manager.cc:1918] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.696239 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 347) took 2159 ms. Trace:
I20250625 13:51:59.696367 1167 rpcz_store.cc:276] 0625 13:51:57.536366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.803596 (+267230us) service_pool.cc:225] Handling call
0625 13:51:59.696043 (+1892447us) ts_tablet_manager.cc:1145] Deleting tablet 27092991fcad475fabc77d1f8bab0582
0625 13:51:59.696055 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.696200 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884625}
I20250625 13:51:59.680881 1166 tablet_bootstrap.cc:492] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.696807 1166 ts_tablet_manager.cc:1397] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.014s
W20250625 13:51:59.696846 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 27092991fcad475fabc77d1f8bab0582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27092991fcad475fabc77d1f8bab0582 already in progress: creating tablet
I20250625 13:51:59.697293 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 349) took 2160 ms. Trace:
I20250625 13:51:59.697399 1167 rpcz_store.cc:276] 0625 13:51:57.536586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.808664 (+272078us) service_pool.cc:225] Handling call
0625 13:51:59.697163 (+1888499us) ts_tablet_manager.cc:1145] Deleting tablet a0e81d4c904248eba858462bcf31c5d9
0625 13:51:59.697171 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.697263 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878605}
W20250625 13:51:59.697783 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a0e81d4c904248eba858462bcf31c5d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e81d4c904248eba858462bcf31c5d9 already in progress: creating tablet
I20250625 13:51:59.697890 1166 raft_consensus.cc:357] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.698104 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 352) took 2114 ms. Trace:
I20250625 13:51:59.698115 1166 raft_consensus.cc:383] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.698215 1166 raft_consensus.cc:738] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.698215 1167 rpcz_store.cc:276] 0625 13:51:57.583270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.814937 (+231667us) service_pool.cc:225] Handling call
0625 13:51:59.697991 (+1883054us) ts_tablet_manager.cc:1145] Deleting tablet 1effed0fa10d409abcb31dbba54861d6
0625 13:51:59.697999 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.698076 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862316}
W20250625 13:51:59.698563 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1effed0fa10d409abcb31dbba54861d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1effed0fa10d409abcb31dbba54861d6 already in progress: creating tablet
I20250625 13:51:59.698757 1166 consensus_queue.cc:260] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:51:59.699441 1166 ts_tablet_manager.cc:1428] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250625 13:51:59.699761 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 353) took 2115 ms. Trace:
W20250625 13:51:59.699891 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 535812d46d6444fda85d9767c6ca7e62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 535812d46d6444fda85d9767c6ca7e62 already in progress: creating tablet
I20250625 13:51:59.699923 1167 rpcz_store.cc:276] 0625 13:51:57.583430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.826521 (+243091us) service_pool.cc:225] Handling call
0625 13:51:59.698845 (+1872324us) ts_tablet_manager.cc:1145] Deleting tablet 02f2a2b595bc43309a29b3f1bc5d3766
0625 13:51:59.698853 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.698925 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1856036}
I20250625 13:51:59.700390 1166 tablet_bootstrap.cc:492] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:51:59.707743 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 02f2a2b595bc43309a29b3f1bc5d3766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02f2a2b595bc43309a29b3f1bc5d3766 already in progress: creating tablet
I20250625 13:51:59.708292 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 364) took 1993 ms. Trace:
I20250625 13:51:59.708632 1167 rpcz_store.cc:276] 0625 13:51:57.714481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.917666 (+203185us) service_pool.cc:225] Handling call
0625 13:51:59.708109 (+1790443us) ts_tablet_manager.cc:1145] Deleting tablet cd4f0857f52449e1b26fde09c6c7c9d4
0625 13:51:59.708121 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.708246 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790248}
W20250625 13:51:59.709303 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet cd4f0857f52449e1b26fde09c6c7c9d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd4f0857f52449e1b26fde09c6c7c9d4 already in progress: creating tablet
I20250625 13:51:59.709795 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 365) took 1893 ms. Trace:
I20250625 13:51:59.709954 1167 rpcz_store.cc:276] 0625 13:51:57.816582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.917871 (+101289us) service_pool.cc:225] Handling call
0625 13:51:59.709653 (+1791782us) ts_tablet_manager.cc:1145] Deleting tablet 32a57268a18f4f00a39f8360f9c90737
0625 13:51:59.709664 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.709760 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1791627}
W20250625 13:51:59.710481 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 32a57268a18f4f00a39f8360f9c90737 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a57268a18f4f00a39f8360f9c90737 already in progress: creating tablet
I20250625 13:51:59.711017 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 366) took 1888 ms. Trace:
I20250625 13:51:59.705542 757 tablet_service.cc:1515] Processing DeleteTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.712883 1167 rpcz_store.cc:276] 0625 13:51:57.822077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.918031 (+ 95954us) service_pool.cc:225] Handling call
0625 13:51:59.710856 (+1792825us) ts_tablet_manager.cc:1145] Deleting tablet 37e4bd4aaab143ffa65a668a952ae345
0625 13:51:59.710869 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.710979 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792709}
I20250625 13:51:59.703083 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 391) took 1374 ms. Trace:
I20250625 13:51:59.713253 1165 rpcz_store.cc:276] 0625 13:51:58.328678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.329146 (+ 468us) service_pool.cc:225] Handling call
0625 13:51:59.702925 (+1373779us) ts_tablet_manager.cc:1145] Deleting tablet 1ca3f3760e6e41a29d004e6afe63ec98
0625 13:51:59.702934 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.703027 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373607}
W20250625 13:51:59.713411 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 37e4bd4aaab143ffa65a668a952ae345 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37e4bd4aaab143ffa65a668a952ae345 already in progress: creating tablet
I20250625 13:51:59.713737 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 392) took 1384 ms. Trace:
I20250625 13:51:59.713853 1165 rpcz_store.cc:276] 0625 13:51:58.328828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.329347 (+ 519us) service_pool.cc:225] Handling call
0625 13:51:59.713561 (+1384214us) ts_tablet_manager.cc:1145] Deleting tablet 7506cce0ec4c4ec68f25591cb98b1252
0625 13:51:59.713570 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.713677 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1384025}
I20250625 13:51:59.714186 1165 tablet_replica.cc:331] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:51:59.714318 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1ca3f3760e6e41a29d004e6afe63ec98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ca3f3760e6e41a29d004e6afe63ec98 already in progress: creating tablet
W20250625 13:51:59.714529 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7506cce0ec4c4ec68f25591cb98b1252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7506cce0ec4c4ec68f25591cb98b1252 already in progress: creating tablet
I20250625 13:51:59.713888 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 367) took 1886 ms. Trace:
I20250625 13:51:59.714826 1167 rpcz_store.cc:276] 0625 13:51:57.827008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.918143 (+ 91135us) service_pool.cc:225] Handling call
0625 13:51:59.713717 (+1795574us) ts_tablet_manager.cc:1145] Deleting tablet 68b8c8aa8036454298bf1e81bab91c01
0625 13:51:59.713726 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.713854 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795431}
I20250625 13:51:59.715420 1166 tablet_bootstrap.cc:654] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:51:59.715574 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 68b8c8aa8036454298bf1e81bab91c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68b8c8aa8036454298bf1e81bab91c01 already in progress: creating tablet
I20250625 13:51:59.716053 1167 tablet_replica.cc:331] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:51:59.716033 1168 log.cc:1199] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/389d81329b8d430a9098b0630612fdcb
I20250625 13:51:59.716539 1168 ts_tablet_manager.cc:1939] T 389d81329b8d430a9098b0630612fdcb P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:59.717465 1167 raft_consensus.cc:2241] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
W20250625 13:51:59.715154 851 consensus_peers.cc:487] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 412a0c5f05ab489ab9f8fc3f44fa7bdc. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:59.697603 1171 tablet_bootstrap.cc:492] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:51:59.718251 1171 ts_tablet_manager.cc:1397] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20250625 13:51:59.719290 1171 raft_consensus.cc:357] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.719641 1171 raft_consensus.cc:383] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.719770 1171 raft_consensus.cc:738] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.720072 1171 consensus_queue.cc:260] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.720728 1171 ts_tablet_manager.cc:1428] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:51:59.726181 1171 tablet_bootstrap.cc:492] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:51:59.729070 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 195) took 4699 ms. Trace:
I20250625 13:51:59.732729 591 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 4 got 0
I20250625 13:51:59.733778 1168 rpcz_store.cc:276] 0625 13:51:55.030085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.031124 (+ 1039us) service_pool.cc:225] Handling call
0625 13:51:59.626635 (+4595511us) ts_tablet_manager.cc:1145] Deleting tablet 389d81329b8d430a9098b0630612fdcb
0625 13:51:59.626646 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.671418 (+ 44772us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.695936 (+ 24518us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.729032 (+ 33096us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4582344}
I20250625 13:51:59.734169 1168 tablet_replica.cc:331] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:51:59.734376 1168 raft_consensus.cc:2241] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.734558 1168 raft_consensus.cc:2270] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.742368 1171 tablet_bootstrap.cc:654] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.745793 589 catalog_manager.cc:5582] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:51:59.747396 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 50786a4c25274ffb8f9910c5dafe3933 (table default.Table_ebb1cd6aa381410a82cac38e7197353c [id=d7cbe338b2554a6e99b07325632aaa33]). Expected version 3 got 0
I20250625 13:51:59.752492 757 tablet_service.cc:1468] Processing CreateTablet for tablet 29a542de77674e33b70647ffd9475fbf (DEFAULT_TABLE table=default.Table_ba042527b71242e8834fd66124edd5a2 [id=ab24ed53e60b48dd86d671d64c454b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.752950 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a542de77674e33b70647ffd9475fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.753863 591 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 4 got 0
I20250625 13:51:59.755092 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 7d361e73fe3d4ea2bb6735f814345ebb from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:59.752485 748 tablet_service.cc:1468] Processing CreateTablet for tablet 25c71f11222e48818ea94984a985d6cb (DEFAULT_TABLE table=default.Table_1f8b426bfe4f426a9840a85d92052857 [id=29c8dc9e4e504fe282078307b9a4a2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.735136 1168 ts_tablet_manager.cc:1905] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.755745 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25c71f11222e48818ea94984a985d6cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.757745 851 consensus_peers.cc:487] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8149aea0ff7646d3b94c9159f56e48ef. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:51:59.758455 753 tablet_service.cc:1468] Processing CreateTablet for tablet 6e8b9b732f8948d2b4c1a81a1de4489a (DEFAULT_TABLE table=default.Table_e54b0d212c3542cc8cb5fb2e997f6be3 [id=325dd21568534ee58900ef0b8d79fae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.758890 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e8b9b732f8948d2b4c1a81a1de4489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.765430 1165 raft_consensus.cc:2241] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.765847 1165 raft_consensus.cc:2270] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.774444 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7d361e73fe3d4ea2bb6735f814345ebb
I20250625 13:51:59.774916 590 catalog_manager.cc:6652] ReplaceTablet: tablet 7d361e73fe3d4ea2bb6735f814345ebb deleted and replaced by tablet 50ac647b1d704142b94da8d506319466
I20250625 13:51:59.775421 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 389d81329b8d430a9098b0630612fdcb (table default.Table_1a06c7c260544cce91ab86dad2d62346 [id=aff513ce6735496784852aef3728d643]) successfully deleted
I20250625 13:51:59.780313 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d361e73fe3d4ea2bb6735f814345ebb with delete_type TABLET_DATA_DELETED (replaced by tablet 50ac647b1d704142b94da8d506319466) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.781258 891 tablet_service.cc:1468] Processing CreateTablet for tablet 29a542de77674e33b70647ffd9475fbf (DEFAULT_TABLE table=default.Table_ba042527b71242e8834fd66124edd5a2 [id=ab24ed53e60b48dd86d671d64c454b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.781688 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a542de77674e33b70647ffd9475fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.783087 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d361e73fe3d4ea2bb6735f814345ebb with delete_type TABLET_DATA_DELETED (replaced by tablet 50ac647b1d704142b94da8d506319466) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.783802 883 tablet_service.cc:1468] Processing CreateTablet for tablet 25c71f11222e48818ea94984a985d6cb (DEFAULT_TABLE table=default.Table_1f8b426bfe4f426a9840a85d92052857 [id=29c8dc9e4e504fe282078307b9a4a2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.784212 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25c71f11222e48818ea94984a985d6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.785449 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 58aa3c21191e4bea9c5395d6e32b8717 from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:59.787082 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e6080511c8eb46c086fdb07159149641" } modify_external_catalogs: true
I20250625 13:51:59.787333 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e6080511c8eb46c086fdb07159149641" } modify_external_catalogs: true
I20250625 13:51:59.787669 1165 ts_tablet_manager.cc:1905] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.788877 1455 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" instance_seqno: 1750859511100740) as {username='slave'} at 127.0.104.3:46495; Asking this server to re-register.
I20250625 13:51:59.789875 1095 heartbeater.cc:461] Registering TS with master...
I20250625 13:51:59.790242 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ddd89f53a2204d2f879f8fd45e20a888"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.790445 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fed63050407b40569f0a8911e9d1fd21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.790871 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fed63050407b40569f0a8911e9d1fd21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.792151 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e22c886829ef45be814505c0f1705376" } modify_external_catalogs: true
I20250625 13:51:59.792322 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e22c886829ef45be814505c0f1705376" } modify_external_catalogs: true
I20250625 13:51:59.792387 1455 ts_manager.cc:194] Registered new tserver with Master: 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:51:59.795117 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_77e602924b364450b34f7bc0c80669f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.795735 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77e602924b364450b34f7bc0c80669f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.790871 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddd89f53a2204d2f879f8fd45e20a888 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.798269 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3f882e0fab474cf2aef02b0067d704f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.798825 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f882e0fab474cf2aef02b0067d704f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.801097 1165 ts_tablet_manager.cc:1918] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:51:59.807250 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4c9efc7631df4ad29112a41f229ae2e9
I20250625 13:51:59.811079 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bb648c45ea344fb3bac765c9e06db2b1
I20250625 13:51:59.814194 1168 ts_tablet_manager.cc:1918] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:51:59.815479 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8603c0f87d5f40dab074f01aaf3d7c2f" } new_table_name: "default.Table_43ed9af0945e4e24b7936931195355f1" modify_external_catalogs: true
I20250625 13:51:59.817715 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ad21d2aa5ffb4443bb2d0b96596545c3" } new_table_name: "default.Table_6df3ba6b40a746ea8650dae9fc9922e8" modify_external_catalogs: true
I20250625 13:51:59.819072 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9efc7631df4ad29112a41f229ae2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.819329 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet bb648c45ea344fb3bac765c9e06db2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.819674 1518 raft_consensus.cc:491] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.819845 1518 raft_consensus.cc:513] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.820593 1518 leader_election.cc:290] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:51:59.821084 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8910e07e05ad4d71bfff7d5a0d6952f4" } modify_external_catalogs: true
I20250625 13:51:59.821251 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8910e07e05ad4d71bfff7d5a0d6952f4" } modify_external_catalogs: true
I20250625 13:51:59.819344 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9efc7631df4ad29112a41f229ae2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.822402 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "600d711975a74f66b9144758ad0dcfb0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:51:59.822497 1168 log.cc:1199] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/32221a413957450fa59ffcfbc91c51b9
I20250625 13:51:59.824488 883 tablet_service.cc:1515] Processing DeleteTablet for tablet bb648c45ea344fb3bac765c9e06db2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.825810 1168 ts_tablet_manager.cc:1939] T 32221a413957450fa59ffcfbc91c51b9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:51:59.828161 1170 tablet_bootstrap.cc:492] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:51:59.828337 1170 ts_tablet_manager.cc:1397] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.155s user 0.004s sys 0.001s
I20250625 13:51:59.829317 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 196) took 4633 ms. Trace:
I20250625 13:51:59.829777 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9766d4ff7d994b58b92a788698344a8a" } new_table_name: "default.Table_2d095d613a32493bbedabb482c8f7d88" modify_external_catalogs: true
I20250625 13:51:59.830081 1170 raft_consensus.cc:357] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.830314 1170 raft_consensus.cc:383] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.830416 1170 raft_consensus.cc:738] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.830462 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ab04e3444c774ad08c56de7ffd7ab607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.830636 1170 consensus_queue.cc:260] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.830767 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_215ad8b5b1b749faa92620b85612d5b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.831073 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab04e3444c774ad08c56de7ffd7ab607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.831132 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_215ad8b5b1b749faa92620b85612d5b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.833835 1170 ts_tablet_manager.cc:1428] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250625 13:51:59.834856 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fe872dae10544c0ca6a6735266cd03d2" } modify_external_catalogs: true
I20250625 13:51:59.835069 1170 tablet_bootstrap.cc:492] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:51:59.835120 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fe872dae10544c0ca6a6735266cd03d2" } modify_external_catalogs: true
W20250625 13:51:59.836113 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10350e0660f3406eaff0ea34b60c47f4 consensus state has no leader
I20250625 13:51:59.835378 1646 raft_consensus.cc:491] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.837407 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 14a12a5453e647b9946ca19e0ffb14ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.837694 1646 raft_consensus.cc:513] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:51:59.838994 1646 leader_election.cc:290] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:59.839217 1168 rpcz_store.cc:276] 0625 13:51:55.196189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.196251 (+ 62us) service_pool.cc:225] Handling call
0625 13:51:59.734105 (+4537854us) ts_tablet_manager.cc:1145] Deleting tablet 32221a413957450fa59ffcfbc91c51b9
0625 13:51:59.734116 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.774076 (+ 39960us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.814063 (+ 39987us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.829261 (+ 15198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4537534}
I20250625 13:51:59.840309 1165 log.cc:1199] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/600d711975a74f66b9144758ad0dcfb0
I20250625 13:51:59.818372 753 tablet_service.cc:1515] Processing DeleteTablet for tablet bb648c45ea344fb3bac765c9e06db2b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.823829 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "600d711975a74f66b9144758ad0dcfb0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:51:59.841014 985 leader_election.cc:343] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: Tablet not RUNNING: STOPPED
I20250625 13:51:59.841154 985 leader_election.cc:304] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:51:59.841445 1518 raft_consensus.cc:2747] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:51:59.839447 849 leader_election.cc:343] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: d0cf3d979369419882a2ba2083933d48
I20250625 13:51:59.810416 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9efc7631df4ad29112a41f229ae2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.841820 1168 tablet_replica.cc:331] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:51:59.841897 851 leader_election.cc:343] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d0cf3d979369419882a2ba2083933d48
W20250625 13:51:59.843210 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35aedb3b590d4917bac8fe0962d68fb3 consensus state has no leader
I20250625 13:51:59.843178 851 leader_election.cc:304] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:51:59.843580 1646 raft_consensus.cc:2747] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:51:59.843994 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet dbd2120b4b084691b6604305773e74f9
I20250625 13:51:59.844260 1168 raft_consensus.cc:2241] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:51:59.844435 1168 raft_consensus.cc:2270] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.845477 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet dbd2120b4b084691b6604305773e74f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.845685 883 tablet_service.cc:1515] Processing DeleteTablet for tablet dbd2120b4b084691b6604305773e74f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.846833 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1653a8a7a93d4e77abf24438181cb8ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.847476 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1653a8a7a93d4e77abf24438181cb8ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.847476 748 tablet_service.cc:1515] Processing DeleteTablet for tablet dbd2120b4b084691b6604305773e74f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.848129 1373 consensus_queue.cc:1035] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:51:59.848412 1165 ts_tablet_manager.cc:1939] T 600d711975a74f66b9144758ad0dcfb0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:51:59.848793 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_46fca0cfc394474781b9a7d01db63df9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:51:59.849323 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46fca0cfc394474781b9a7d01db63df9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.849627 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 393) took 1513 ms. Trace:
I20250625 13:51:59.849754 1165 rpcz_store.cc:276] 0625 13:51:58.336342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.336398 (+ 56us) service_pool.cc:225] Handling call
0625 13:51:59.714096 (+1377698us) ts_tablet_manager.cc:1145] Deleting tablet 600d711975a74f66b9144758ad0dcfb0
0625 13:51:59.714106 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.765670 (+ 51564us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x6160000c1354. stack: 00007f4161830336 00007f41618300f0 00007f41608263ad 0000000000971004 0000000000970fcd 000000000097099a 00007f416d290000 00007f416e5a2093 00007f417107f137 00007f41710b11c4 00007f4171092c20 00007f41710929ad 0000000000a7573c 00007f41618759df 00007f4161878cab 00007f4161878b28
0625 13:51:59.793105 (+ 27435us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.800948 (+ 7843us) tablet_metadata.cc:641] Metadata flushed
0625 13:51:59.849591 (+ 48643us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117335936,"tablet-delete.queue_time_us":1377481}
I20250625 13:51:59.850214 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 600d711975a74f66b9144758ad0dcfb0 (table default.Table_186697102c744ac39ec9191fc0f992ea [id=548110492fba410e9612ddd9792eed22]) successfully deleted
I20250625 13:51:59.851063 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0089084e327b47a1a45bf11be8539647"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.851451 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4cf923b8b24141eeab3c245e08664637"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:51:59.851501 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b83d984846e24954a5702dc7392d94b1" } new_table_name: "default.Table_899784ce715e4e229d6d804c9a94cbcd" modify_external_catalogs: true
W20250625 13:51:59.851992 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cf923b8b24141eeab3c245e08664637 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:51:59.852731 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0089084e327b47a1a45bf11be8539647 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:51:59.852980 1168 ts_tablet_manager.cc:1905] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.853468 1047 raft_consensus.cc:3058] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:51:59.854128 1167 raft_consensus.cc:2270] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.854238 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_96a7158ed5964f8aaff3f72fc357f629" } new_table_name: "default.Table_9c6d8799477b4081b2652a4df3b13762" modify_external_catalogs: true
I20250625 13:51:59.771562 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 25c71f11222e48818ea94984a985d6cb (DEFAULT_TABLE table=default.Table_1f8b426bfe4f426a9840a85d92052857 [id=29c8dc9e4e504fe282078307b9a4a2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.854983 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25c71f11222e48818ea94984a985d6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.772043 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 8149aea0ff7646d3b94c9159f56e48ef with delete_type TABLET_DATA_DELETED (replaced by tablet 88f72c555d9842f28ca4bf310da9c53c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.857404 883 tablet_service.cc:1468] Processing CreateTablet for tablet 6e8b9b732f8948d2b4c1a81a1de4489a (DEFAULT_TABLE table=default.Table_e54b0d212c3542cc8cb5fb2e997f6be3 [id=325dd21568534ee58900ef0b8d79fae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.857884 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e8b9b732f8948d2b4c1a81a1de4489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.861385 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_517319d59c234fe9a5fe83530d3a00c6" } modify_external_catalogs: true
I20250625 13:51:59.861569 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_517319d59c234fe9a5fe83530d3a00c6" } modify_external_catalogs: true
I20250625 13:51:59.772349 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 29a542de77674e33b70647ffd9475fbf (DEFAULT_TABLE table=default.Table_ba042527b71242e8834fd66124edd5a2 [id=ab24ed53e60b48dd86d671d64c454b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.774019 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 36da1da539e24f55978d516873cb0b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.774811 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 6e8b9b732f8948d2b4c1a81a1de4489a (DEFAULT_TABLE table=default.Table_e54b0d212c3542cc8cb5fb2e997f6be3 [id=325dd21568534ee58900ef0b8d79fae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.864336 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e8b9b732f8948d2b4c1a81a1de4489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.864836 1166 tablet_bootstrap.cc:492] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:51:59.866019 748 tablet_service.cc:1468] Processing CreateTablet for tablet fa16fa387b2a4bb5899b1487642c48d0 (DEFAULT_TABLE table=default.Table_1493c363105b457a9b8b6a7673039e85 [id=eda56c372c514507aeaa7e578fcdf099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.867055 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa16fa387b2a4bb5899b1487642c48d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.775919 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d361e73fe3d4ea2bb6735f814345ebb with delete_type TABLET_DATA_DELETED (replaced by tablet 50ac647b1d704142b94da8d506319466) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.867650 753 tablet_service.cc:1468] Processing CreateTablet for tablet 5f49c0fc97d44b28acfd6cb493906edb (DEFAULT_TABLE table=default.Table_0965fa983950458882df05ba9d414275 [id=3b49ef69287446cd8df199820b9c60c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.867905 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a542de77674e33b70647ffd9475fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.868005 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f49c0fc97d44b28acfd6cb493906edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.871403 1166 ts_tablet_manager.cc:1397] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.166s user 0.012s sys 0.000s
I20250625 13:51:59.872925 1014 tablet_service.cc:1468] Processing CreateTablet for tablet fa16fa387b2a4bb5899b1487642c48d0 (DEFAULT_TABLE table=default.Table_1493c363105b457a9b8b6a7673039e85 [id=eda56c372c514507aeaa7e578fcdf099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.873291 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa16fa387b2a4bb5899b1487642c48d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.875638 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 5f49c0fc97d44b28acfd6cb493906edb (DEFAULT_TABLE table=default.Table_0965fa983950458882df05ba9d414275 [id=3b49ef69287446cd8df199820b9c60c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.876045 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f49c0fc97d44b28acfd6cb493906edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.876977 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 389) took 1552 ms. Trace:
I20250625 13:51:59.877139 1165 rpcz_store.cc:276] 0625 13:51:58.324691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.370945 (+ 46254us) service_pool.cc:225] Handling call
0625 13:51:59.876786 (+1505841us) ts_tablet_manager.cc:1145] Deleting tablet d43ba17ebfec4244894d1b6e23200c33
0625 13:51:59.876796 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.876937 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1504922}
W20250625 13:51:59.877651 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d43ba17ebfec4244894d1b6e23200c33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43ba17ebfec4244894d1b6e23200c33 already in progress: creating tablet
I20250625 13:51:59.878607 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 396) took 1268 ms. Trace:
I20250625 13:51:59.878736 1165 rpcz_store.cc:276] 0625 13:51:58.610549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.610603 (+ 54us) service_pool.cc:225] Handling call
0625 13:51:59.878457 (+1267854us) ts_tablet_manager.cc:1145] Deleting tablet 9be75d03121f4ea2906236ac3a5b0764
0625 13:51:59.878467 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.878572 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256761}
W20250625 13:51:59.879196 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9be75d03121f4ea2906236ac3a5b0764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9be75d03121f4ea2906236ac3a5b0764 already in progress: creating tablet
I20250625 13:51:59.879644 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 405) took 1076 ms. Trace:
I20250625 13:51:59.879760 1165 rpcz_store.cc:276] 0625 13:51:58.802973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.803136 (+ 163us) service_pool.cc:225] Handling call
0625 13:51:59.879491 (+1076355us) ts_tablet_manager.cc:1145] Deleting tablet 055f88e118454e32baa578cae9aceb0c
0625 13:51:59.879500 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.879610 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074281}
W20250625 13:51:59.880213 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 055f88e118454e32baa578cae9aceb0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 055f88e118454e32baa578cae9aceb0c already in progress: creating tablet
I20250625 13:51:59.880609 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 408) took 1074 ms. Trace:
I20250625 13:51:59.880712 1165 rpcz_store.cc:276] 0625 13:51:58.805969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.806085 (+ 116us) service_pool.cc:225] Handling call
0625 13:51:59.880441 (+1074356us) ts_tablet_manager.cc:1145] Deleting tablet 0b129117db064a15b27e82439637cf22
0625 13:51:59.880449 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.880578 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028926}
W20250625 13:51:59.881090 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0b129117db064a15b27e82439637cf22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b129117db064a15b27e82439637cf22 already in progress: creating tablet
I20250625 13:51:59.879262 1166 raft_consensus.cc:357] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.881367 1165 tablet_replica.cc:331] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:51:59.881722 1165 raft_consensus.cc:2241] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:51:59.882397 1166 raft_consensus.cc:383] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:51:59.882252 757 tablet_service.cc:1468] Processing CreateTablet for tablet 34371795c8094e73ab4e5dbeedb7aa2f (DEFAULT_TABLE table=default.Table_cc3ae4d4f62d47af9a510dab1dd32ff5 [id=092db20955194462a83605209ee2ce2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.883446 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34371795c8094e73ab4e5dbeedb7aa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.883402 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 34371795c8094e73ab4e5dbeedb7aa2f (DEFAULT_TABLE table=default.Table_cc3ae4d4f62d47af9a510dab1dd32ff5 [id=092db20955194462a83605209ee2ce2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.883785 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34371795c8094e73ab4e5dbeedb7aa2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.886066 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d89b0ddcc14148faa1a6926fcbb027a8 consensus state has no leader
I20250625 13:51:59.888514 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 (DEFAULT_TABLE table=default.Table_ee7ffdafe40d48bc89ffa9a18595cf9d [id=abe1686e960e477f9ce60dbcd6a23ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.888926 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17efe97d559a45dc9c0ca0b74efa7aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.888983 753 tablet_service.cc:1468] Processing CreateTablet for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 (DEFAULT_TABLE table=default.Table_ee7ffdafe40d48bc89ffa9a18595cf9d [id=abe1686e960e477f9ce60dbcd6a23ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.889350 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17efe97d559a45dc9c0ca0b74efa7aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.890900 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet dcf7fab53cae49908d2952cdaa07943a
I20250625 13:51:59.891817 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf7fab53cae49908d2952cdaa07943a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.891880 756 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf7fab53cae49908d2952cdaa07943a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.892839 1166 raft_consensus.cc:738] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:51:59.882241 1165 pending_rounds.cc:70] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:51:59.894411 1165 pending_rounds.cc:77] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520590180777984 op_type: NO_OP noop_request { }
I20250625 13:51:59.894557 1373 raft_consensus.cc:491] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.894799 1373 raft_consensus.cc:513] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.896617 1165 raft_consensus.cc:2887] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:51:59.896922 1027 tablet_service.cc:1468] Processing CreateTablet for tablet dc1c9cea3d5a405a958c6c7585dbef39 (DEFAULT_TABLE table=default.Table_722e0f5afe094da88dd5c6985558f1c1 [id=822e163e6c7344e9b49b8d3164212c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.897315 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1c9cea3d5a405a958c6c7585dbef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.907553 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 3016763b9e294b9da2861fe97e57e241 (DEFAULT_TABLE table=default.Table_21f2f615829242bfb6460e42b5995d14 [id=0f2c02514ee0468b9aaa8095831374ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.907953 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3016763b9e294b9da2861fe97e57e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.911649 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 4d25e4114a184bc99705042593ab4ab9 (DEFAULT_TABLE table=default.Table_1452b75693a442568546fb6e23ff895e [id=7031f07a2e03441da6bd1aff4f907060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.912051 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b4be0bc23cde4f0f94bd678096e0c7d8 (DEFAULT_TABLE table=default.Table_da32ce790d1e43469d2ece95ae454446 [id=42418562238e4d25af2c9db41c25633a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.913959 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 10350e0660f3406eaff0ea34b60c47f4 (DEFAULT_TABLE table=default.Table_43ed9af0945e4e24b7936931195355f1 [id=b39790d2326440edbf0249b8115791b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.914281 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d85cdadfd694486590092a7b6f61f44c (DEFAULT_TABLE table=default.Table_4d8e666a80a345f7ba2ad05b84ce5a05 [id=560d105a82934a11a344edfee582205c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.914746 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4be0bc23cde4f0f94bd678096e0c7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.914963 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10350e0660f3406eaff0ea34b60c47f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.915405 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 150af49427ae47b98e9b2077bbbe1e2b (DEFAULT_TABLE table=default.Table_9ce2182835d94395bb248b8d70c733d8 [id=177d2a03fddb4d1b8a396df46c4b9650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.915783 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 150af49427ae47b98e9b2077bbbe1e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.919740 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85cdadfd694486590092a7b6f61f44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.920526 1604 raft_consensus.cc:491] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:51:59.920775 1604 raft_consensus.cc:513] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.718467 1652 tablet.cc:1722] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:51:59.924098 1167 ts_tablet_manager.cc:1905] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.900849 756 tablet_service.cc:1468] Processing CreateTablet for tablet 4d25e4114a184bc99705042593ab4ab9 (DEFAULT_TABLE table=default.Table_1452b75693a442568546fb6e23ff895e [id=7031f07a2e03441da6bd1aff4f907060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.906212 758 tablet_service.cc:1468] Processing CreateTablet for tablet 150af49427ae47b98e9b2077bbbe1e2b (DEFAULT_TABLE table=default.Table_9ce2182835d94395bb248b8d70c733d8 [id=177d2a03fddb4d1b8a396df46c4b9650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.913259 755 tablet_service.cc:1468] Processing CreateTablet for tablet 10350e0660f3406eaff0ea34b60c47f4 (DEFAULT_TABLE table=default.Table_43ed9af0945e4e24b7936931195355f1 [id=b39790d2326440edbf0249b8115791b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.922681 1373 leader_election.cc:290] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:51:59.925926 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af05fba6fe2042aab5dd47b30e4dfcf3 consensus state has no leader
I20250625 13:51:59.925484 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d25e4114a184bc99705042593ab4ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.919656 1028 tablet_service.cc:1468] Processing CreateTablet for tablet fb4973cbffc743dba98d7b7c9f76be3f (DEFAULT_TABLE table=default.Table_6820dad35d484763a041822f9b7bd57f [id=a208b3c34fb748459dcb266c82afbf60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.929359 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4973cbffc743dba98d7b7c9f76be3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.929531 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3016763b9e294b9da2861fe97e57e241 (DEFAULT_TABLE table=default.Table_21f2f615829242bfb6460e42b5995d14 [id=0f2c02514ee0468b9aaa8095831374ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.930565 749 tablet_service.cc:1468] Processing CreateTablet for tablet fb4973cbffc743dba98d7b7c9f76be3f (DEFAULT_TABLE table=default.Table_6820dad35d484763a041822f9b7bd57f [id=a208b3c34fb748459dcb266c82afbf60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.930778 1027 tablet_service.cc:1468] Processing CreateTablet for tablet d89b0ddcc14148faa1a6926fcbb027a8 (DEFAULT_TABLE table=default.Table_2d095d613a32493bbedabb482c8f7d88 [id=d5578088aed44652808da11ed52404dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.931061 752 tablet_service.cc:1468] Processing CreateTablet for tablet b4be0bc23cde4f0f94bd678096e0c7d8 (DEFAULT_TABLE table=default.Table_da32ce790d1e43469d2ece95ae454446 [id=42418562238e4d25af2c9db41c25633a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.931529 745 tablet_service.cc:1468] Processing CreateTablet for tablet d85cdadfd694486590092a7b6f61f44c (DEFAULT_TABLE table=default.Table_4d8e666a80a345f7ba2ad05b84ce5a05 [id=560d105a82934a11a344edfee582205c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.932013 746 tablet_service.cc:1468] Processing CreateTablet for tablet ed19663d21d84c9e90d4f4dda055420d (DEFAULT_TABLE table=default.Table_984c484e0c964d3889777159c75343a7 [id=281623c0feb944b9bf59149c146aa64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.932134 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 077e8c001cc646559b5268d4a48c0ad6
W20250625 13:51:59.932240 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 063a793e0d7747668d2728fdc2160b5e consensus state has no leader
I20250625 13:51:59.932317 750 tablet_service.cc:1468] Processing CreateTablet for tablet d89b0ddcc14148faa1a6926fcbb027a8 (DEFAULT_TABLE table=default.Table_2d095d613a32493bbedabb482c8f7d88 [id=d5578088aed44652808da11ed52404dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.931859 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89b0ddcc14148faa1a6926fcbb027a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.933004 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:51:59.933455 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d25e4114a184bc99705042593ab4ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.933737 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3016763b9e294b9da2861fe97e57e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.933923 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85cdadfd694486590092a7b6f61f44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.934187 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10350e0660f3406eaff0ea34b60c47f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.934386 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 150af49427ae47b98e9b2077bbbe1e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.946471 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4be0bc23cde4f0f94bd678096e0c7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.949664 1165 raft_consensus.cc:2270] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:51:59.950059 1604 leader_election.cc:290] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:51:59.951072 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c911d9ecbded417daf70e85c7cbcd52e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:51:59.951349 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4973cbffc743dba98d7b7c9f76be3f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.951627 711 leader_election.cc:343] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:51:59.929536 1166 consensus_queue.cc:260] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:51:59.952407 1166 ts_tablet_manager.cc:1428] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
I20250625 13:51:59.952749 1166 tablet_bootstrap.cc:492] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:51:59.961225 849 leader_election.cc:343] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 7fe053e294fe4aceb24919e9e2bddf65
I20250625 13:51:59.971832 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 8b7062c34fd143bf86b743055a27db8c from {username='slave'} at 127.0.0.1:54762
I20250625 13:51:59.974992 1165 ts_tablet_manager.cc:1905] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:51:59.976351 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 (DEFAULT_TABLE table=default.Table_e90d64fdf05e442aa9da241d5fa1263b [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.976558 756 tablet_service.cc:1468] Processing CreateTablet for tablet a8334d8187d2473582f0057193874b74 (DEFAULT_TABLE table=default.Table_4dfad83f70604649b15f116a1d9aa9c5 [id=f7c0d014047943bb9d33256b24a98fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.975898 757 tablet_service.cc:1468] Processing CreateTablet for tablet dc1c9cea3d5a405a958c6c7585dbef39 (DEFAULT_TABLE table=default.Table_722e0f5afe094da88dd5c6985558f1c1 [id=822e163e6c7344e9b49b8d3164212c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.979110 1166 tablet_bootstrap.cc:654] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:51:59.979455 891 tablet_service.cc:1468] Processing CreateTablet for tablet fa16fa387b2a4bb5899b1487642c48d0 (DEFAULT_TABLE table=default.Table_1493c363105b457a9b8b6a7673039e85 [id=eda56c372c514507aeaa7e578fcdf099]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.979300 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89b0ddcc14148faa1a6926fcbb027a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.979941 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa16fa387b2a4bb5899b1487642c48d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.980713 884 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf7fab53cae49908d2952cdaa07943a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:51:59.981178 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed19663d21d84c9e90d4f4dda055420d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.981763 886 tablet_service.cc:1468] Processing CreateTablet for tablet 34371795c8094e73ab4e5dbeedb7aa2f (DEFAULT_TABLE table=default.Table_cc3ae4d4f62d47af9a510dab1dd32ff5 [id=092db20955194462a83605209ee2ce2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.982034 741 tablet_service.cc:1468] Processing CreateTablet for tablet 0f177c0da5ab41a784a512d6eeb48d29 (DEFAULT_TABLE table=default.Table_279ed5effac34385a5eb2ea2439ec1ae [id=095583ecc61145248d019934ecc6ed6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.982637 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0998ad87fcc145999f6430e0de7b1304 (DEFAULT_TABLE table=default.Table_407b8d22e01548a4aadf917bd73fa8f6 [id=2a13920984524c03b88636288f8570ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.983110 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a8334d8187d2473582f0057193874b74 (DEFAULT_TABLE table=default.Table_4dfad83f70604649b15f116a1d9aa9c5 [id=f7c0d014047943bb9d33256b24a98fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.983129 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 0f177c0da5ab41a784a512d6eeb48d29 (DEFAULT_TABLE table=default.Table_279ed5effac34385a5eb2ea2439ec1ae [id=095583ecc61145248d019934ecc6ed6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.983563 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8334d8187d2473582f0057193874b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.983510 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 (DEFAULT_TABLE table=default.Table_e90d64fdf05e442aa9da241d5fa1263b [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.983908 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0998ad87fcc145999f6430e0de7b1304 (DEFAULT_TABLE table=default.Table_407b8d22e01548a4aadf917bd73fa8f6 [id=2a13920984524c03b88636288f8570ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.984144 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.984344 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:51:59.984395 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.984647 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1c9cea3d5a405a958c6c7585dbef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.985997 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f177c0da5ab41a784a512d6eeb48d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.987805 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8334d8187d2473582f0057193874b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:51:59.988615 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0998ad87fcc145999f6430e0de7b1304. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:51:59.995147 851 leader_election.cc:343] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7fe053e294fe4aceb24919e9e2bddf65
I20250625 13:51:59.996255 892 tablet_service.cc:1468] Processing CreateTablet for tablet 5f49c0fc97d44b28acfd6cb493906edb (DEFAULT_TABLE table=default.Table_0965fa983950458882df05ba9d414275 [id=3b49ef69287446cd8df199820b9c60c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:51:59.998976 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 7efc8df3c8e24020b073a1b61bc9e0a0 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.000034 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34371795c8094e73ab4e5dbeedb7aa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.003999 1166 tablet_bootstrap.cc:492] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:00.004336 1166 ts_tablet_manager.cc:1397] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.004s
I20250625 13:52:00.008002 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7efc8df3c8e24020b073a1b61bc9e0a0
I20250625 13:52:00.008491 588 catalog_manager.cc:6652] ReplaceTablet: tablet 7efc8df3c8e24020b073a1b61bc9e0a0 deleted and replaced by tablet 3f06030f45284c5a9929a4c4513004ba
I20250625 13:52:00.008352 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_70ec24bdd5764fa1893a92a1e341a2ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.008945 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70ec24bdd5764fa1893a92a1e341a2ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.010054 851 leader_election.cc:304] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:00.010411 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 52470.8us
W20250625 13:52:00.010699 851 consensus_peers.cc:487] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d524089e6ab044ec961cfc9f587e8f86. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:00.011008 1604 raft_consensus.cc:2747] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:00.011770 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dbca3ace8835418e8db2cbc1845d9cfb" } modify_external_catalogs: true
I20250625 13:52:00.011957 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dbca3ace8835418e8db2cbc1845d9cfb" } modify_external_catalogs: true
I20250625 13:52:00.012782 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9ce2182835d94395bb248b8d70c733d8" } modify_external_catalogs: true
I20250625 13:52:00.012975 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9ce2182835d94395bb248b8d70c733d8" } modify_external_catalogs: true
I20250625 13:52:00.013204 1373 raft_consensus.cc:491] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.013362 1373 raft_consensus.cc:513] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.013535 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f49c0fc97d44b28acfd6cb493906edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.013940 887 tablet_service.cc:1468] Processing CreateTablet for tablet dc1c9cea3d5a405a958c6c7585dbef39 (DEFAULT_TABLE table=default.Table_722e0f5afe094da88dd5c6985558f1c1 [id=822e163e6c7344e9b49b8d3164212c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.014472 884 tablet_service.cc:1468] Processing CreateTablet for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 (DEFAULT_TABLE table=default.Table_ee7ffdafe40d48bc89ffa9a18595cf9d [id=abe1686e960e477f9ce60dbcd6a23ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.019560 895 tablet_service.cc:1468] Processing CreateTablet for tablet 150af49427ae47b98e9b2077bbbe1e2b (DEFAULT_TABLE table=default.Table_9ce2182835d94395bb248b8d70c733d8 [id=177d2a03fddb4d1b8a396df46c4b9650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.019815 877 tablet_service.cc:1468] Processing CreateTablet for tablet 3016763b9e294b9da2861fe97e57e241 (DEFAULT_TABLE table=default.Table_21f2f615829242bfb6460e42b5995d14 [id=0f2c02514ee0468b9aaa8095831374ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.020871 880 tablet_service.cc:1468] Processing CreateTablet for tablet 4d25e4114a184bc99705042593ab4ab9 (DEFAULT_TABLE table=default.Table_1452b75693a442568546fb6e23ff895e [id=7031f07a2e03441da6bd1aff4f907060]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.021519 879 tablet_service.cc:1468] Processing CreateTablet for tablet fb4973cbffc743dba98d7b7c9f76be3f (DEFAULT_TABLE table=default.Table_6820dad35d484763a041822f9b7bd57f [id=a208b3c34fb748459dcb266c82afbf60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.029265 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_35471795fa7d4688ae966d55ca175371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.029901 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35471795fa7d4688ae966d55ca175371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.030280 1168 ts_tablet_manager.cc:1918] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.030412 1168 log.cc:1199] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d0cf3d979369419882a2ba2083933d48
I20250625 13:52:00.031991 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f04ee0e6b3324e169e99b807c1faa3a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.032500 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f04ee0e6b3324e169e99b807c1faa3a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.033064 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 46316eb725bc47e695af476c0b185504 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.035032 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 155737a8d5604eb69524c4d6bea61550 (DEFAULT_TABLE table=default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.035593 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 8a780f2d2daf465a940d364477139b73 (DEFAULT_TABLE table=default.Table_075afd49a8144b2eab9d29164f19a5a3 [id=9978514e7a5845e78f5071b59fa5c5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.036167 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ff65e23c44624b2cbf92c2dc67eaed09 (DEFAULT_TABLE table=default.Table_f683f5d47e2c4c708cbe9cbbcca71ee6 [id=03f4f845af2941209df88d6c95070aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.036733 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f06030f45284c5a9929a4c4513004ba) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.037325 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_20830d90ca194d2d8a472b90228ae5b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.037864 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20830d90ca194d2d8a472b90228ae5b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.038379 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_215ad8b5b1b749faa92620b85612d5b9" } new_table_name: "default.Table_1fc704a30db04ed98f279746d4f875b2" modify_external_catalogs: true
I20250625 13:52:00.033397 1025 tablet_service.cc:1468] Processing CreateTablet for tablet ed19663d21d84c9e90d4f4dda055420d (DEFAULT_TABLE table=default.Table_984c484e0c964d3889777159c75343a7 [id=281623c0feb944b9bf59149c146aa64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.041271 885 tablet_service.cc:1468] Processing CreateTablet for tablet d85cdadfd694486590092a7b6f61f44c (DEFAULT_TABLE table=default.Table_4d8e666a80a345f7ba2ad05b84ce5a05 [id=560d105a82934a11a344edfee582205c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.043335 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 829221b8a79f460d90dd24be6a2d4c43
I20250625 13:52:00.043953 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 150af49427ae47b98e9b2077bbbe1e2b
I20250625 13:52:00.046499 889 tablet_service.cc:1468] Processing CreateTablet for tablet d89b0ddcc14148faa1a6926fcbb027a8 (DEFAULT_TABLE table=default.Table_2d095d613a32493bbedabb482c8f7d88 [id=d5578088aed44652808da11ed52404dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.046999 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ab04e3444c774ad08c56de7ffd7ab607" } modify_external_catalogs: true
I20250625 13:52:00.047180 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ab04e3444c774ad08c56de7ffd7ab607" } modify_external_catalogs: true
I20250625 13:52:00.047847 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6bb95cabd7b940cdafc8a4978beeac70" } new_table_name: "default.Table_fac2050c0ed04e3da584967a45644198" modify_external_catalogs: true
I20250625 13:52:00.049049 896 tablet_service.cc:1468] Processing CreateTablet for tablet b4be0bc23cde4f0f94bd678096e0c7d8 (DEFAULT_TABLE table=default.Table_da32ce790d1e43469d2ece95ae454446 [id=42418562238e4d25af2c9db41c25633a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:00.049558 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc350f8573f74dd88dcbc4c9a7c192c4 has no consensus state
I20250625 13:52:00.050117 882 tablet_service.cc:1468] Processing CreateTablet for tablet 10350e0660f3406eaff0ea34b60c47f4 (DEFAULT_TABLE table=default.Table_43ed9af0945e4e24b7936931195355f1 [id=b39790d2326440edbf0249b8115791b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.050419 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17efe97d559a45dc9c0ca0b74efa7aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.050400 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 0f63e9037bd74f1eb39bd85aa820ec35 (DEFAULT_TABLE table=default.Table_f02770cf5ed24271b290c8224d3675d6 [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.050640 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f177c0da5ab41a784a512d6eeb48d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.051128 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d25e4114a184bc99705042593ab4ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.053483 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed19663d21d84c9e90d4f4dda055420d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.054683 1518 consensus_queue.cc:1035] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250625 13:52:00.055334 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 150af49427ae47b98e9b2077bbbe1e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.055557 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85cdadfd694486590092a7b6f61f44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.055907 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4973cbffc743dba98d7b7c9f76be3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.056696 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet b187643aca6e47d6a0f72ae932c2e92d
I20250625 13:52:00.056852 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4be0bc23cde4f0f94bd678096e0c7d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.057178 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3016763b9e294b9da2861fe97e57e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.057561 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10350e0660f3406eaff0ea34b60c47f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.058302 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89b0ddcc14148faa1a6926fcbb027a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.058501 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1c9cea3d5a405a958c6c7585dbef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.059904 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155737a8d5604eb69524c4d6bea61550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.060328 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.061430 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.066191 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_550cc88382f845bcad839c4bcd436274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.066774 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_550cc88382f845bcad839c4bcd436274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.070042 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 150af49427ae47b98e9b2077bbbe1e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:00.070930 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eafc402d809245d189f3977162506ef8 consensus state has no leader
I20250625 13:52:00.073873 1373 leader_election.cc:290] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:00.052953 1168 ts_tablet_manager.cc:1939] T d0cf3d979369419882a2ba2083933d48 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:00.085181 752 tablet_service.cc:1468] Processing CreateTablet for tablet 8a780f2d2daf465a940d364477139b73 (DEFAULT_TABLE table=default.Table_075afd49a8144b2eab9d29164f19a5a3 [id=9978514e7a5845e78f5071b59fa5c5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.085623 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a780f2d2daf465a940d364477139b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.088558 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f73f420a4d7a438eadde3d6af64f0a11" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.088585 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9234ef66a80b476b9614ca31733dfa74" } new_table_name: "default.Table_467fcbf2d0ba44108a7aa82294efb342" modify_external_catalogs: true
I20250625 13:52:00.091028 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0998ad87fcc145999f6430e0de7b1304. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:00.097146 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ae5f1d414794474b2178b431cbc1d44 consensus state has no leader
I20250625 13:52:00.099939 1170 tablet_bootstrap.cc:654] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.103279 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a780f2d2daf465a940d364477139b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.103979 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f63e9037bd74f1eb39bd85aa820ec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.105322 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff65e23c44624b2cbf92c2dc67eaed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.105770 758 tablet_service.cc:1468] Processing CreateTablet for tablet ff65e23c44624b2cbf92c2dc67eaed09 (DEFAULT_TABLE table=default.Table_f683f5d47e2c4c708cbe9cbbcca71ee6 [id=03f4f845af2941209df88d6c95070aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.111536 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f683f5d47e2c4c708cbe9cbbcca71ee6" } modify_external_catalogs: true
I20250625 13:52:00.111732 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f683f5d47e2c4c708cbe9cbbcca71ee6" } modify_external_catalogs: true
I20250625 13:52:00.113955 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_56560af1ef3144d4a8daed436f4b0505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.114476 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56560af1ef3144d4a8daed436f4b0505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.121354 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ff65e23c44624b2cbf92c2dc67eaed09
I20250625 13:52:00.125190 590 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd]). Expected version 4 got 3
I20250625 13:52:00.134505 1652 tablet.cc:1717] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Skipping requested alter to schema version 3, tablet already version 3
I20250625 13:52:00.137214 1652 tablet.cc:1722] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:00.140259 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_914aeef494b1417b96fd83b69c631c69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:00.140446 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_05bee7b38e4a49e99da18772c39769d7" } new_table_name: "default.Table_cc09be6f62454883abfeda8b152ab65f" modify_external_catalogs: true
W20250625 13:52:00.140955 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_914aeef494b1417b96fd83b69c631c69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.142408 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a091a655459e4488b9f5d5798673fd42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.142894 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a091a655459e4488b9f5d5798673fd42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.144030 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_21175492fe5f4223a50a60ebf15306c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.144462 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21175492fe5f4223a50a60ebf15306c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.145641 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7d1b445e65344381aac8680dad76a4b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.146111 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d1b445e65344381aac8680dad76a4b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.147110 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_340984100eaf438ab7fb71232d7fe268" } new_table_name: "default.Table_d0f7159508314d49992e34cf8af1e00c" modify_external_catalogs: true
I20250625 13:52:00.148288 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f2851488b3864c7b893b63c84369288d" } modify_external_catalogs: true
I20250625 13:52:00.148459 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f2851488b3864c7b893b63c84369288d" } modify_external_catalogs: true
W20250625 13:52:00.153057 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c3f0ca6fa5482db7f57143911c838f consensus state has no leader
W20250625 13:52:00.157428 711 leader_election.cc:343] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:00.157608 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 68411.5us
I20250625 13:52:00.158295 753 tablet_service.cc:1468] Processing CreateTablet for tablet 155737a8d5604eb69524c4d6bea61550 (DEFAULT_TABLE table=default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:00.158505 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dad4ea8cf44944b08f53428c95809daf consensus state has no leader
I20250625 13:52:00.158919 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cbf7e30cdff245208dacc6963fe88c7d
I20250625 13:52:00.159010 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f06030f45284c5a9929a4c4513004ba) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.158982 1166 raft_consensus.cc:357] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.159504 1166 raft_consensus.cc:383] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.159060 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.159761 1049 raft_consensus.cc:3058] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:00.159760 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0f63e9037bd74f1eb39bd85aa820ec35 (DEFAULT_TABLE table=default.Table_f02770cf5ed24271b290c8224d3675d6 [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.160205 1166 raft_consensus.cc:738] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.160799 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff65e23c44624b2cbf92c2dc67eaed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.163396 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 150af49427ae47b98e9b2077bbbe1e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.163751 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f63e9037bd74f1eb39bd85aa820ec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.167407 1166 consensus_queue.cc:260] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.170835 1166 ts_tablet_manager.cc:1428] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.166s user 0.003s sys 0.000s
I20250625 13:52:00.171512 1166 tablet_bootstrap.cc:492] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:00.183743 747 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf7e30cdff245208dacc6963fe88c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.187291 1518 raft_consensus.cc:491] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.187502 1518 raft_consensus.cc:513] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.188151 1518 leader_election.cc:290] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:00.188817 985 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 36da1da539e24f55978d516873cb0b27
I20250625 13:52:00.190477 1166 tablet_bootstrap.cc:654] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.191992 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet ff65e23c44624b2cbf92c2dc67eaed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.196434 895 tablet_service.cc:1468] Processing CreateTablet for tablet ed19663d21d84c9e90d4f4dda055420d (DEFAULT_TABLE table=default.Table_984c484e0c964d3889777159c75343a7 [id=281623c0feb944b9bf59149c146aa64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.209049 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 197) took 4996 ms. Trace:
W20250625 13:52:00.210026 849 consensus_peers.cc:487] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:00.211527 1646 consensus_queue.cc:1035] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20250625 13:52:00.212147 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed19663d21d84c9e90d4f4dda055420d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.215438 1168 rpcz_store.cc:276] 0625 13:51:55.199245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.200353 (+ 1108us) service_pool.cc:225] Handling call
0625 13:51:59.841723 (+4641370us) ts_tablet_manager.cc:1145] Deleting tablet d0cf3d979369419882a2ba2083933d48
0625 13:51:59.841737 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:51:59.959964 (+118227us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.030131 (+ 70167us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.195642 (+165511us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4641154}
I20250625 13:52:00.219668 1167 ts_tablet_manager.cc:1918] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.224707 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36da1da539e24f55978d516873cb0b27" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:00.225306 881 tablet_service.cc:1468] Processing CreateTablet for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 (DEFAULT_TABLE table=default.Table_e90d64fdf05e442aa9da241d5fa1263b [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.230185 912 raft_consensus.cc:2466] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:00.230520 883 tablet_service.cc:1468] Processing CreateTablet for tablet a8334d8187d2473582f0057193874b74 (DEFAULT_TABLE table=default.Table_4dfad83f70604649b15f116a1d9aa9c5 [id=f7c0d014047943bb9d33256b24a98fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.233173 878 tablet_service.cc:1468] Processing CreateTablet for tablet 0998ad87fcc145999f6430e0de7b1304 (DEFAULT_TABLE table=default.Table_407b8d22e01548a4aadf917bd73fa8f6 [id=2a13920984524c03b88636288f8570ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.239079 893 tablet_service.cc:1468] Processing CreateTablet for tablet 0f177c0da5ab41a784a512d6eeb48d29 (DEFAULT_TABLE table=default.Table_279ed5effac34385a5eb2ea2439ec1ae [id=095583ecc61145248d019934ecc6ed6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.239339 894 tablet_service.cc:1468] Processing CreateTablet for tablet 155737a8d5604eb69524c4d6bea61550 (DEFAULT_TABLE table=default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.242918 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8a780f2d2daf465a940d364477139b73 (DEFAULT_TABLE table=default.Table_075afd49a8144b2eab9d29164f19a5a3 [id=9978514e7a5845e78f5071b59fa5c5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.250377 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c7c83caf0473417da3a8fc96412d92f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.251114 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7c83caf0473417da3a8fc96412d92f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.253885 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet d0cf3d979369419882a2ba2083933d48 (table default.Table_ac5c66505c43484ca1990febfc2a5bf9 [id=eb254e00e07047a09c1924a6cab20e4d]) successfully deleted
I20250625 13:52:00.254551 1510 raft_consensus.cc:491] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.254698 1510 raft_consensus.cc:513] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.231700 983 leader_election.cc:304] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:00.256922 1167 log.cc:1199] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a21335a06ec04da29b3351554c75f03c
I20250625 13:52:00.257247 1518 raft_consensus.cc:2802] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:00.257375 1518 raft_consensus.cc:491] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:00.257491 1518 raft_consensus.cc:3058] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:00.258347 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_63e7af63093f48dd8b6e416d8676fade" } modify_external_catalogs: true
I20250625 13:52:00.258411 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fc57b2c273df4835bd06effe454ccc27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:00.258553 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_63e7af63093f48dd8b6e416d8676fade" } modify_external_catalogs: true
W20250625 13:52:00.259021 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc57b2c273df4835bd06effe454ccc27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.259528 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_984c484e0c964d3889777159c75343a7" } modify_external_catalogs: true
I20250625 13:52:00.259701 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_984c484e0c964d3889777159c75343a7" } modify_external_catalogs: true
I20250625 13:52:00.260213 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ba042527b71242e8834fd66124edd5a2" } new_table_name: "default.Table_fe2ddd54af1b4192a1378dca82cfcca5" modify_external_catalogs: true
W20250625 13:52:00.242718 985 consensus_peers.cc:487] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07e281786e6e4a5d93b94b85b486655d. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:00.260980 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_44d60eefccee45558969e859fa7897c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:00.257501 1167 ts_tablet_manager.cc:1939] T a21335a06ec04da29b3351554c75f03c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:00.261586 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44d60eefccee45558969e859fa7897c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.263008 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_472e8cfcd53c4be2b738c172ae9fb892"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.263559 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_472e8cfcd53c4be2b738c172ae9fb892 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.264690 1518 raft_consensus.cc:513] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.265007 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 371) took 2321 ms. Trace:
I20250625 13:52:00.265373 1518 leader_election.cc:290] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:00.265153 1167 rpcz_store.cc:276] 0625 13:51:57.940430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.009372 (+ 68942us) service_pool.cc:225] Handling call
0625 13:51:59.715982 (+1706610us) ts_tablet_manager.cc:1145] Deleting tablet a21335a06ec04da29b3351554c75f03c
0625 13:51:59.715993 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.065357 (+349364us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.219492 (+154135us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.262014 (+ 42522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676446}
I20250625 13:52:00.267269 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a21335a06ec04da29b3351554c75f03c (table default.Table_1329da083f314d9d8fa3ac2e851646bf [id=3286dc94d1ba4645beb4a2a54bec8c8a]) successfully deleted
I20250625 13:52:00.267889 1167 tablet_replica.cc:331] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.268222 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9c6d8799477b4081b2652a4df3b13762" } modify_external_catalogs: true
I20250625 13:52:00.268373 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9c6d8799477b4081b2652a4df3b13762" } modify_external_catalogs: true
W20250625 13:52:00.269107 985 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 36da1da539e24f55978d516873cb0b27
I20250625 13:52:00.269752 1167 raft_consensus.cc:2241] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
W20250625 13:52:00.273450 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29a542de77674e33b70647ffd9475fbf consensus state has no leader
I20250625 13:52:00.274053 593 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 568abed285dd4d43809a3062c41b766d
I20250625 13:52:00.274387 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ed19663d21d84c9e90d4f4dda055420d
I20250625 13:52:00.284792 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_21f2f615829242bfb6460e42b5995d14" } new_table_name: "default.Table_dfa5acc20d2d4b34ba0a3503952923f5" modify_external_catalogs: true
I20250625 13:52:00.285957 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.286629 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:00.291021 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3016763b9e294b9da2861fe97e57e241 consensus state has no leader
I20250625 13:52:00.298585 1510 leader_election.cc:290] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:00.270200 1170 tablet_bootstrap.cc:492] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:00.295398 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet ed19663d21d84c9e90d4f4dda055420d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.299623 1170 ts_tablet_manager.cc:1397] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.465s user 0.057s sys 0.049s
I20250625 13:52:00.299801 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41a7862601f04fd7a938f7e5d5a4c0f9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.300469 1170 raft_consensus.cc:357] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.300765 1170 raft_consensus.cc:383] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.300900 1170 raft_consensus.cc:738] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.301153 1170 consensus_queue.cc:260] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.301724 1170 ts_tablet_manager.cc:1428] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:00.302008 1170 tablet_bootstrap.cc:492] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:00.302975 849 leader_election.cc:343] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.303947 1167 pending_rounds.cc:70] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:00.304080 1167 pending_rounds.cc:77] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520586355982336 op_type: NO_OP noop_request { }
I20250625 13:52:00.304266 1167 raft_consensus.cc:2887] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:00.304443 1167 raft_consensus.cc:2270] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.304844 1170 tablet_bootstrap.cc:654] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.305210 1167 ts_tablet_manager.cc:1905] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.309275 1652 tablet.cc:1722] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:00.321297 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_80f2c7cdc1504002b60af89a706bec5c" } new_table_name: "default.Table_1c969e628596444b93540412505c815a" modify_external_catalogs: true
I20250625 13:52:00.324919 1171 tablet_bootstrap.cc:492] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:00.325089 1171 ts_tablet_manager.cc:1397] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.599s user 0.006s sys 0.005s
I20250625 13:52:00.325881 1171 raft_consensus.cc:357] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.326095 1171 raft_consensus.cc:383] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.326179 1171 raft_consensus.cc:738] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.326382 1171 consensus_queue.cc:260] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.326946 1171 ts_tablet_manager.cc:1428] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:00.327725 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36da1da539e24f55978d516873cb0b27" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:00.327909 914 raft_consensus.cc:3058] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:00.328116 888 tablet_service.cc:1468] Processing CreateTablet for tablet ff65e23c44624b2cbf92c2dc67eaed09 (DEFAULT_TABLE table=default.Table_f683f5d47e2c4c708cbe9cbbcca71ee6 [id=03f4f845af2941209df88d6c95070aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.329699 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f73f420a4d7a438eadde3d6af64f0a11" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:00.330533 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c911d9ecbded417daf70e85c7cbcd52e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:00.330801 914 raft_consensus.cc:2466] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:00.330922 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:00.332115 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c1deb52366c404cbdc08076d4bcb3d1 consensus state has no leader
I20250625 13:52:00.332234 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f06030f45284c5a9929a4c4513004ba) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.332419 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.332567 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 150af49427ae47b98e9b2077bbbe1e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.332717 880 tablet_service.cc:1515] Processing DeleteTablet for tablet ff65e23c44624b2cbf92c2dc67eaed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.333163 880 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf7e30cdff245208dacc6963fe88c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.333329 880 tablet_service.cc:1515] Processing DeleteTablet for tablet ed19663d21d84c9e90d4f4dda055420d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:00.333860 849 consensus_peers.cc:487] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:00.331115 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 200) took 4992 ms. Trace:
I20250625 13:52:00.335422 1168 rpcz_store.cc:276] 0625 13:51:55.338522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.338570 (+ 48us) service_pool.cc:225] Handling call
0625 13:52:00.215784 (+4877214us) ts_tablet_manager.cc:1145] Deleting tablet 04479dfb34584f0891c604ef72bf93b5
0625 13:52:00.330973 (+115189us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.331074 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4876994}
I20250625 13:52:00.335858 1168 tablet_replica.cc:331] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:00.331696 896 tablet_service.cc:1468] Processing CreateTablet for tablet 0f63e9037bd74f1eb39bd85aa820ec35 (DEFAULT_TABLE table=default.Table_f02770cf5ed24271b290c8224d3675d6 [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.336511 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f63e9037bd74f1eb39bd85aa820ec35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.342851 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_07eaa93fdfa1484f98c567581341f0e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.343503 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07eaa93fdfa1484f98c567581341f0e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.344816 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_21175492fe5f4223a50a60ebf15306c1" } modify_external_catalogs: true
I20250625 13:52:00.345326 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_21175492fe5f4223a50a60ebf15306c1" } modify_external_catalogs: true
W20250625 13:52:00.349342 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 04479dfb34584f0891c604ef72bf93b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04479dfb34584f0891c604ef72bf93b5 already in progress: creating tablet
I20250625 13:52:00.334656 1167 ts_tablet_manager.cc:1918] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.253350 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf7e30cdff245208dacc6963fe88c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.349756 1167 log.cc:1199] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/67441795e22b445a90c88d76aa519ea5
I20250625 13:52:00.350224 1167 ts_tablet_manager.cc:1939] T 67441795e22b445a90c88d76aa519ea5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.350461 591 master_service.cc:948] ReplaceTablet: received request to replace tablet efc9536a716f4b9f9fc4af1637b3cd80 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.350592 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 373) took 2191 ms. Trace:
I20250625 13:52:00.350682 1167 rpcz_store.cc:276] 0625 13:51:58.158756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.159374 (+ 618us) service_pool.cc:225] Handling call
0625 13:52:00.267807 (+2108433us) ts_tablet_manager.cc:1145] Deleting tablet 67441795e22b445a90c88d76aa519ea5
0625 13:52:00.267817 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.311356 (+ 43539us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.334499 (+ 23143us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.350529 (+ 16030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2107924}
I20250625 13:52:00.351133 1171 tablet_bootstrap.cc:492] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:00.351125 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 372) took 2192 ms. Trace:
I20250625 13:52:00.351326 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0998ad87fcc145999f6430e0de7b1304. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.351212 1167 rpcz_store.cc:276] 0625 13:51:58.158298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.163039 (+ 4741us) service_pool.cc:225] Handling call
0625 13:52:00.350964 (+2187925us) ts_tablet_manager.cc:1145] Deleting tablet db6e9db2375144b38ec43ffca9ca4c1e
0625 13:52:00.350973 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.351073 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2187565}
I20250625 13:52:00.351745 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 382) took 1976 ms. Trace:
I20250625 13:52:00.351837 1167 rpcz_store.cc:276] 0625 13:51:58.375265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.375580 (+ 315us) service_pool.cc:225] Handling call
0625 13:52:00.351607 (+1976027us) ts_tablet_manager.cc:1145] Deleting tablet d43ba17ebfec4244894d1b6e23200c33
0625 13:52:00.351615 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.351696 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926654}
I20250625 13:52:00.352166 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 383) took 1976 ms. Trace:
I20250625 13:52:00.352243 1167 rpcz_store.cc:276] 0625 13:51:58.375397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.375600 (+ 203us) service_pool.cc:225] Handling call
0625 13:52:00.352032 (+1976432us) ts_tablet_manager.cc:1145] Deleting tablet 535812d46d6444fda85d9767c6ca7e62
0625 13:52:00.352040 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.352123 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926146}
I20250625 13:52:00.353065 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 384) took 1936 ms. Trace:
I20250625 13:52:00.353144 1167 rpcz_store.cc:276] 0625 13:51:58.416256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.416436 (+ 180us) service_pool.cc:225] Handling call
0625 13:52:00.352416 (+1935980us) ts_tablet_manager.cc:1145] Deleting tablet 1ca3f3760e6e41a29d004e6afe63ec98
0625 13:52:00.352424 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.353017 (+ 593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1913889}
I20250625 13:52:00.353457 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 385) took 1931 ms. Trace:
I20250625 13:52:00.353533 1167 rpcz_store.cc:276] 0625 13:51:58.422186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.429388 (+ 7202us) service_pool.cc:225] Handling call
0625 13:52:00.353338 (+1923950us) ts_tablet_manager.cc:1145] Deleting tablet 7506cce0ec4c4ec68f25591cb98b1252
0625 13:52:00.353346 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.353417 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1875804}
I20250625 13:52:00.353865 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 387) took 1754 ms. Trace:
I20250625 13:52:00.353956 1167 rpcz_store.cc:276] 0625 13:51:58.599264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.599399 (+ 135us) service_pool.cc:225] Handling call
0625 13:52:00.353721 (+1754322us) ts_tablet_manager.cc:1145] Deleting tablet 9be75d03121f4ea2906236ac3a5b0764
0625 13:52:00.353730 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.353823 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1754084}
I20250625 13:52:00.354200 1171 tablet_bootstrap.cc:654] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.354218 1167 tablet_replica.cc:331] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.354398 1167 raft_consensus.cc:2241] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.354564 1167 raft_consensus.cc:2270] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.355073 1167 ts_tablet_manager.cc:1905] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:00.356503 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet db6e9db2375144b38ec43ffca9ca4c1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db6e9db2375144b38ec43ffca9ca4c1e already in progress: creating tablet
W20250625 13:52:00.356806 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d43ba17ebfec4244894d1b6e23200c33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43ba17ebfec4244894d1b6e23200c33 already in progress: creating tablet
W20250625 13:52:00.357019 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 535812d46d6444fda85d9767c6ca7e62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 535812d46d6444fda85d9767c6ca7e62 already in progress: creating tablet
W20250625 13:52:00.357237 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1ca3f3760e6e41a29d004e6afe63ec98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ca3f3760e6e41a29d004e6afe63ec98 already in progress: creating tablet
W20250625 13:52:00.357442 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7506cce0ec4c4ec68f25591cb98b1252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7506cce0ec4c4ec68f25591cb98b1252 already in progress: creating tablet
W20250625 13:52:00.357735 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9be75d03121f4ea2906236ac3a5b0764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9be75d03121f4ea2906236ac3a5b0764 already in progress: creating tablet
I20250625 13:52:00.344404 983 leader_election.cc:304] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:00.359395 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff65e23c44624b2cbf92c2dc67eaed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.359688 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a780f2d2daf465a940d364477139b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.359735 1518 raft_consensus.cc:2802] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:00.363364 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet efc9536a716f4b9f9fc4af1637b3cd80
I20250625 13:52:00.363464 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155737a8d5604eb69524c4d6bea61550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.363834 591 catalog_manager.cc:6652] ReplaceTablet: tablet efc9536a716f4b9f9fc4af1637b3cd80 deleted and replaced by tablet fd4142572b344456a0d8248be914ad8f
I20250625 13:52:00.364665 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 631f98dffffa4267a6c0604ced0d183b
I20250625 13:52:00.365221 1518 raft_consensus.cc:695] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:00.365777 1518 consensus_queue.cc:237] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.367470 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8334d8187d2473582f0057193874b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.371555 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f177c0da5ab41a784a512d6eeb48d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.375430 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet efc9536a716f4b9f9fc4af1637b3cd80 with delete_type TABLET_DATA_DELETED (replaced by tablet fd4142572b344456a0d8248be914ad8f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.390229 893 tablet_service.cc:1515] Processing DeleteTablet for tablet efc9536a716f4b9f9fc4af1637b3cd80 with delete_type TABLET_DATA_DELETED (replaced by tablet fd4142572b344456a0d8248be914ad8f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.392596 1510 raft_consensus.cc:491] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.392822 1510 raft_consensus.cc:513] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.393777 1167 ts_tablet_manager.cc:1918] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.393914 1167 log.cc:1199] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/4b549bc8cead4af4a72f41f51655602c
I20250625 13:52:00.394116 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "799f2edd960d45c1ace9195283a042e2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.394367 1167 ts_tablet_manager.cc:1939] T 4b549bc8cead4af4a72f41f51655602c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.394461 1510 leader_election.cc:290] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:00.394685 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 388) took 1677 ms. Trace:
W20250625 13:52:00.394884 849 leader_election.cc:343] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.394795 1167 rpcz_store.cc:276] 0625 13:51:58.717093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.717149 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:00.354135 (+1636986us) ts_tablet_manager.cc:1145] Deleting tablet 4b549bc8cead4af4a72f41f51655602c
0625 13:52:00.354142 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.374237 (+ 20095us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.393629 (+ 19392us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.394647 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636701}
I20250625 13:52:00.395219 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 4b549bc8cead4af4a72f41f51655602c (table default.Table_e02e9aad565644299f4613c5b807162c [id=27271698cfa246659d7f91010c8752d9]) successfully deleted
I20250625 13:52:00.395975 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 397) took 1637 ms. Trace:
I20250625 13:52:00.396159 1167 rpcz_store.cc:276] 0625 13:51:58.758511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.758585 (+ 74us) service_pool.cc:225] Handling call
0625 13:52:00.395839 (+1637254us) ts_tablet_manager.cc:1145] Deleting tablet 055f88e118454e32baa578cae9aceb0c
0625 13:52:00.395848 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.395944 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1637015}
W20250625 13:52:00.396708 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 055f88e118454e32baa578cae9aceb0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 055f88e118454e32baa578cae9aceb0c already in progress: creating tablet
I20250625 13:52:00.397222 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 398) took 1637 ms. Trace:
I20250625 13:52:00.397363 1167 rpcz_store.cc:276] 0625 13:51:58.759425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.759492 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:00.397103 (+1637611us) ts_tablet_manager.cc:1145] Deleting tablet d9462da7574a46839763b1c388102e60
0625 13:52:00.397112 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.397184 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1637395}
W20250625 13:52:00.397840 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d9462da7574a46839763b1c388102e60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9462da7574a46839763b1c388102e60 already in progress: creating tablet
I20250625 13:52:00.398415 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 400) took 1635 ms. Trace:
I20250625 13:52:00.398839 1167 rpcz_store.cc:276] 0625 13:51:58.763053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.763107 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:00.398279 (+1635172us) ts_tablet_manager.cc:1145] Deleting tablet 0b129117db064a15b27e82439637cf22
0625 13:52:00.398288 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.398381 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1634964}
W20250625 13:52:00.399751 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0b129117db064a15b27e82439637cf22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b129117db064a15b27e82439637cf22 already in progress: creating tablet
I20250625 13:52:00.400274 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 399) took 1637 ms. Trace:
W20250625 13:52:00.400504 851 consensus_peers.cc:487] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02f2a2b595bc43309a29b3f1bc5d3766. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:00.400817 1167 rpcz_store.cc:276] 0625 13:51:58.762916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.763368 (+ 452us) service_pool.cc:225] Handling call
0625 13:52:00.400135 (+1636767us) ts_tablet_manager.cc:1145] Deleting tablet 3249c5c2aa564027a2c594ca4c31e1c8
0625 13:52:00.400144 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.400230 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636562}
W20250625 13:52:00.401448 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3249c5c2aa564027a2c594ca4c31e1c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3249c5c2aa564027a2c594ca4c31e1c8 already in progress: creating tablet
I20250625 13:52:00.401906 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 401) took 1628 ms. Trace:
I20250625 13:52:00.402360 1167 rpcz_store.cc:276] 0625 13:51:58.773418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.773485 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:00.401755 (+1628270us) ts_tablet_manager.cc:1145] Deleting tablet 38a23b08a03b48eaa051f670eb7dac65
0625 13:52:00.401765 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.401873 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627993}
W20250625 13:52:00.403023 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 38a23b08a03b48eaa051f670eb7dac65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38a23b08a03b48eaa051f670eb7dac65 already in progress: creating tablet
I20250625 13:52:00.403556 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 402) took 1628 ms. Trace:
I20250625 13:52:00.403962 1167 rpcz_store.cc:276] 0625 13:51:58.775091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.775179 (+ 88us) service_pool.cc:225] Handling call
0625 13:52:00.403441 (+1628262us) ts_tablet_manager.cc:1145] Deleting tablet a30a35e3639a43c5b078169060824723
0625 13:52:00.403449 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.403515 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628036}
W20250625 13:52:00.404593 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a30a35e3639a43c5b078169060824723 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a30a35e3639a43c5b078169060824723 already in progress: creating tablet
I20250625 13:52:00.405050 1167 tablet_replica.cc:331] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.405442 1167 raft_consensus.cc:2241] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.405858 1167 raft_consensus.cc:2270] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.406963 1167 ts_tablet_manager.cc:1905] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.178807 1165 ts_tablet_manager.cc:1918] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.408084 1165 log.cc:1199] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/bd5ba76c130a4711af5b21e9eb8562e7
I20250625 13:52:00.408612 1165 ts_tablet_manager.cc:1939] T bd5ba76c130a4711af5b21e9eb8562e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:00.409020 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 416) took 1557 ms. Trace:
I20250625 13:52:00.409225 1165 rpcz_store.cc:276] 0625 13:51:58.851867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.852099 (+ 232us) service_pool.cc:225] Handling call
0625 13:51:59.881307 (+1029208us) ts_tablet_manager.cc:1145] Deleting tablet bd5ba76c130a4711af5b21e9eb8562e7
0625 13:51:59.881314 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.139685 (+258371us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.178658 (+ 38973us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.408986 (+230328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019356}
I20250625 13:52:00.409955 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet bd5ba76c130a4711af5b21e9eb8562e7 (table default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]) successfully deleted
I20250625 13:52:00.410435 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 407) took 1604 ms. Trace:
I20250625 13:52:00.410642 1165 rpcz_store.cc:276] 0625 13:51:58.805834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.808559 (+ 2725us) service_pool.cc:225] Handling call
0625 13:52:00.410253 (+1601694us) ts_tablet_manager.cc:1145] Deleting tablet 3249c5c2aa564027a2c594ca4c31e1c8
0625 13:52:00.410262 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.410379 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548091}
W20250625 13:52:00.411123 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3249c5c2aa564027a2c594ca4c31e1c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3249c5c2aa564027a2c594ca4c31e1c8 already in progress: creating tablet
I20250625 13:52:00.411528 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 410) took 1604 ms. Trace:
I20250625 13:52:00.411700 1165 rpcz_store.cc:276] 0625 13:51:58.806636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.807076 (+ 440us) service_pool.cc:225] Handling call
0625 13:52:00.411404 (+1604328us) ts_tablet_manager.cc:1145] Deleting tablet a30a35e3639a43c5b078169060824723
0625 13:52:00.411413 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.411499 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541549}
W20250625 13:52:00.412273 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a30a35e3639a43c5b078169060824723 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a30a35e3639a43c5b078169060824723 already in progress: creating tablet
I20250625 13:52:00.412649 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 406) took 1607 ms. Trace:
I20250625 13:52:00.412797 1165 rpcz_store.cc:276] 0625 13:51:58.805327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.805597 (+ 270us) service_pool.cc:225] Handling call
0625 13:52:00.412526 (+1606929us) ts_tablet_manager.cc:1145] Deleting tablet d9462da7574a46839763b1c388102e60
0625 13:52:00.412534 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.412620 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541048}
W20250625 13:52:00.413300 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d9462da7574a46839763b1c388102e60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9462da7574a46839763b1c388102e60 already in progress: creating tablet
I20250625 13:52:00.413659 1165 tablet_replica.cc:331] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:00.223503 1373 raft_consensus.cc:491] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.413875 1373 raft_consensus.cc:513] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.414611 1373 leader_election.cc:290] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:00.256399 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41a7862601f04fd7a938f7e5d5a4c0f9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:00.415150 776 raft_consensus.cc:2466] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:00.267629 747 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.284875 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ed19663d21d84c9e90d4f4dda055420d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.343966 777 raft_consensus.cc:1273] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer affa1c3035b940e88ca143be7d35840d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:00.366212 740 tablet_service.cc:1515] Processing DeleteTablet for tablet efc9536a716f4b9f9fc4af1637b3cd80 with delete_type TABLET_DATA_DELETED (replaced by tablet fd4142572b344456a0d8248be914ad8f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.396024 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "799f2edd960d45c1ace9195283a042e2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:00.418709 851 leader_election.cc:304] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:00.419359 1646 raft_consensus.cc:2802] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:00.419474 1646 raft_consensus.cc:491] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:00.419555 1646 raft_consensus.cc:3058] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:00.421684 1167 ts_tablet_manager.cc:1918] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.421836 1167 log.cc:1199] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/2c1a98dc40684954a602567cf3476879
I20250625 13:52:00.422612 1646 raft_consensus.cc:513] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.422819 1167 ts_tablet_manager.cc:1939] T 2c1a98dc40684954a602567cf3476879 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.423148 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 406) took 1495 ms. Trace:
I20250625 13:52:00.423751 1167 rpcz_store.cc:276] 0625 13:51:58.927818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.980513 (+ 52695us) service_pool.cc:225] Handling call
0625 13:52:00.404988 (+1424475us) ts_tablet_manager.cc:1145] Deleting tablet 2c1a98dc40684954a602567cf3476879
0625 13:52:00.404996 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.412951 (+ 7955us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.419846 (+ 6895us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.423081 (+ 3235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418376}
I20250625 13:52:00.424326 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41a7862601f04fd7a938f7e5d5a4c0f9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:00.424456 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 2c1a98dc40684954a602567cf3476879 (table default.Table_1231c5f6367b4046beaa57696ea42ff4 [id=97fc76973f074d29903fa25a80ca534a]) successfully deleted
W20250625 13:52:00.424830 849 leader_election.cc:343] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.424971 1167 tablet_replica.cc:331] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.425077 1646 leader_election.cc:290] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:00.425148 1167 raft_consensus.cc:2241] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.425318 1167 raft_consensus.cc:2270] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.425560 1646 consensus_queue.cc:1035] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:00.425796 1167 ts_tablet_manager.cc:1905] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.427435 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41a7862601f04fd7a938f7e5d5a4c0f9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:00.427652 777 raft_consensus.cc:3058] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:00.430734 777 raft_consensus.cc:2466] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:00.432598 851 leader_election.cc:304] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:00.434005 1646 raft_consensus.cc:2802] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:00.434280 1646 raft_consensus.cc:695] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:00.434589 1646 consensus_queue.cc:237] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.437400 1510 raft_consensus.cc:491] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.437657 1510 raft_consensus.cc:513] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.439625 1167 ts_tablet_manager.cc:1918] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.440452 1510 leader_election.cc:290] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:00.441305 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c72f9ae92a4537979fb479eb4467c4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:00.441439 1167 log.cc:1199] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/cbaa018753d8455199f3869323fb5582
I20250625 13:52:00.441569 776 raft_consensus.cc:2466] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:00.441854 1167 ts_tablet_manager.cc:1939] T cbaa018753d8455199f3869323fb5582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.442284 851 leader_election.cc:304] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:00.440068 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c72f9ae92a4537979fb479eb4467c4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.442793 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 407) took 1514 ms. Trace:
W20250625 13:52:00.443399 849 leader_election.cc:343] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.443413 1167 rpcz_store.cc:276] 0625 13:51:58.927936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.981075 (+ 53139us) service_pool.cc:225] Handling call
0625 13:52:00.424892 (+1443817us) ts_tablet_manager.cc:1145] Deleting tablet cbaa018753d8455199f3869323fb5582
0625 13:52:00.424900 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.433363 (+ 8463us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.439453 (+ 6090us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.442753 (+ 3300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438155}
I20250625 13:52:00.443845 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet cbaa018753d8455199f3869323fb5582 (table default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]) successfully deleted
I20250625 13:52:00.443962 1510 raft_consensus.cc:2802] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:00.444067 1510 raft_consensus.cc:491] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:00.444216 1510 raft_consensus.cc:3058] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:00.444900 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 408) took 1511 ms. Trace:
I20250625 13:52:00.445190 1167 rpcz_store.cc:276] 0625 13:51:58.933537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.023869 (+ 90332us) service_pool.cc:225] Handling call
0625 13:52:00.444423 (+1420554us) ts_tablet_manager.cc:1145] Deleting tablet ccce48011a15405db2b95c04c9b01b3c
0625 13:52:00.444431 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.444868 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420005}
W20250625 13:52:00.445566 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_cc09be6f62454883abfeda8b152ab65f [id=82115aa29a8f456b8d7c2f3d9caa17e1], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet c7c3f0ca6fa5482db7f57143911c838f consensus state has no leader
W20250625 13:52:00.446317 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ccce48011a15405db2b95c04c9b01b3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccce48011a15405db2b95c04c9b01b3c already in progress: creating tablet
I20250625 13:52:00.446717 1167 tablet_replica.cc:331] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.447157 1167 raft_consensus.cc:2241] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.447737 1167 raft_consensus.cc:2270] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.448592 1510 raft_consensus.cc:513] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.448926 1167 ts_tablet_manager.cc:1905] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.398128 748 tablet_service.cc:1515] Processing DeleteTablet for tablet ff65e23c44624b2cbf92c2dc67eaed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.190774 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 155737a8d5604eb69524c4d6bea61550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.454041 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1c969e628596444b93540412505c815a" } new_table_name: "default.Table_b2b0087eeea54823809b847648141d81" modify_external_catalogs: true
I20250625 13:52:00.454041 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_688372ad7a204de3aa17a0fcf7c0a34f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.455082 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_688372ad7a204de3aa17a0fcf7c0a34f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.455528 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ecb211456c1e44e08827d9509ba961d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.456046 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecb211456c1e44e08827d9509ba961d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.458269 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_467fcbf2d0ba44108a7aa82294efb342" } modify_external_catalogs: true
I20250625 13:52:00.458453 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_467fcbf2d0ba44108a7aa82294efb342" } modify_external_catalogs: true
I20250625 13:52:00.458588 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f02770cf5ed24271b290c8224d3675d6" } new_table_name: "default.Table_1099b7a2614540d3ac476e1bad30554c" modify_external_catalogs: true
I20250625 13:52:00.459365 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c72f9ae92a4537979fb479eb4467c4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:00.459829 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c9c75aecd94048888373fbb8cc3efdbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.460294 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9c75aecd94048888373fbb8cc3efdbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:00.460920 849 leader_election.cc:343] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.461093 1510 leader_election.cc:290] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:00.461481 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_72f464f3d7e7440da8df36ec84428a8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:00.461709 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c72f9ae92a4537979fb479eb4467c4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
W20250625 13:52:00.461906 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72f464f3d7e7440da8df36ec84428a8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.461948 776 raft_consensus.cc:3058] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:00.464920 776 raft_consensus.cc:2466] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:00.465668 851 leader_election.cc:304] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
W20250625 13:52:00.465965 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c1deb52366c404cbdc08076d4bcb3d1 consensus state has no leader
I20250625 13:52:00.466504 1442 raft_consensus.cc:2802] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:00.467274 1442 raft_consensus.cc:695] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:00.467629 1442 consensus_queue.cc:237] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.477718 1167 ts_tablet_manager.cc:1918] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:00.481014 1167 log.cc:1199] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d4f19ddc51324cc18242c6081f3f7de5
I20250625 13:52:00.482905 1167 ts_tablet_manager.cc:1939] T d4f19ddc51324cc18242c6081f3f7de5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.483481 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 410) took 1445 ms. Trace:
I20250625 13:52:00.483484 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ba34195043ab4dd79dea215f57cbb3cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:00.484979 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b5fe7dfea5c84fc4a7cc420c24d1e54b" } new_table_name: "default.Table_9bc32b4aab234a4e8bab3910a4a3102b" modify_external_catalogs: true
I20250625 13:52:00.486970 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94" } modify_external_catalogs: true
I20250625 13:52:00.487150 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94" } modify_external_catalogs: true
I20250625 13:52:00.488819 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5c8024face7142ac823c087d60f8b7b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.489423 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c8024face7142ac823c087d60f8b7b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.491209 1167 rpcz_store.cc:276] 0625 13:51:59.037907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.037961 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:00.446650 (+1408689us) ts_tablet_manager.cc:1145] Deleting tablet d4f19ddc51324cc18242c6081f3f7de5
0625 13:52:00.446659 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.469158 (+ 22499us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.477550 (+ 8392us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.483422 (+ 5872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1404094}
I20250625 13:52:00.491544 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8cdfd947725147e294c585373c3c4fc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.492076 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cdfd947725147e294c585373c3c4fc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.492389 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet d4f19ddc51324cc18242c6081f3f7de5 (table default.Table_08c36a3d2a03417982751548b18def03 [id=b8285b64bbee49f9be8741b252d03363]) successfully deleted
I20250625 13:52:00.493134 1167 tablet_replica.cc:331] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.493566 1167 raft_consensus.cc:2241] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
W20250625 13:52:00.484201 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba34195043ab4dd79dea215f57cbb3cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.494290 1167 pending_rounds.cc:70] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:00.494722 1167 pending_rounds.cc:77] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520590050676736 op_type: NO_OP noop_request { }
I20250625 13:52:00.496019 1167 raft_consensus.cc:2887] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:00.496366 1167 raft_consensus.cc:2270] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.497316 1167 ts_tablet_manager.cc:1905] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:00.507583 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f63e9037bd74f1eb39bd85aa820ec35 consensus state has no leader
I20250625 13:52:00.511402 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1956bf8628234a2fa1b29b76594f624b" } modify_external_catalogs: true
I20250625 13:52:00.511638 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1956bf8628234a2fa1b29b76594f624b" } modify_external_catalogs: true
I20250625 13:52:00.513628 1167 ts_tablet_manager.cc:1918] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.515668 1167 log.cc:1199] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/07e281786e6e4a5d93b94b85b486655d
I20250625 13:52:00.516361 1167 ts_tablet_manager.cc:1939] T 07e281786e6e4a5d93b94b85b486655d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.517292 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 409) took 1479 ms. Trace:
I20250625 13:52:00.517891 1167 rpcz_store.cc:276] 0625 13:51:59.037783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.038197 (+ 414us) service_pool.cc:225] Handling call
0625 13:52:00.492773 (+1454576us) ts_tablet_manager.cc:1145] Deleting tablet 07e281786e6e4a5d93b94b85b486655d
0625 13:52:00.492784 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.506283 (+ 13499us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.513500 (+ 7217us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.517239 (+ 3739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450140}
I20250625 13:52:00.518785 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 07e281786e6e4a5d93b94b85b486655d (table default.Table_fb311c42df024421b325596e0cb7a778 [id=7b67603755e7428496524ee716b435e9]) successfully deleted
I20250625 13:52:00.521724 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1f9d1b5465174503915b895238d69e85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:00.519249 1167 tablet_replica.cc:331] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:00.522347 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f9d1b5465174503915b895238d69e85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.535771 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cc09be6f62454883abfeda8b152ab65f" } new_table_name: "default.Table_9b2f5c3f9f4d4ea69644097de7596642" modify_external_catalogs: true
I20250625 13:52:00.545948 1168 raft_consensus.cc:2241] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.546339 1168 pending_rounds.cc:70] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:00.546433 1168 pending_rounds.cc:77] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520590050676736 op_type: NO_OP noop_request { }
I20250625 13:52:00.546561 1168 raft_consensus.cc:2887] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:00.546686 1168 raft_consensus.cc:2270] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:00.549494 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f4d6be016284dc796945270178b9ee1 has no consensus state
I20250625 13:52:00.555837 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6a3e9fc3400f4884a44a9054a64afe1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.557885 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a3e9fc3400f4884a44a9054a64afe1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.560575 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet afa1c198dd6541809843244c7cc18194
I20250625 13:52:00.562485 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet afa1c198dd6541809843244c7cc18194 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.566457 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7e3cd23ead3f4afa889078106d9b4711"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.567225 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e3cd23ead3f4afa889078106d9b4711 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.567816 886 tablet_service.cc:1515] Processing DeleteTablet for tablet afa1c198dd6541809843244c7cc18194 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.569072 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_77e602924b364450b34f7bc0c80669f5" } new_table_name: "default.Table_f766c58c1858460cbad7a7f98f1ac55e" modify_external_catalogs: true
I20250625 13:52:00.581385 1510 raft_consensus.cc:491] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.581777 1510 raft_consensus.cc:513] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.582770 1510 leader_election.cc:290] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:00.584895 1167 raft_consensus.cc:2241] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.585502 1167 raft_consensus.cc:2270] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.593035 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9bb7ebc40a14ecaa0c692a8e1117db3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.597229 1167 ts_tablet_manager.cc:1905] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.605551 1167 ts_tablet_manager.cc:1918] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.605822 1167 log.cc:1199] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/600d711975a74f66b9144758ad0dcfb0
I20250625 13:52:00.606321 1167 ts_tablet_manager.cc:1939] T 600d711975a74f66b9144758ad0dcfb0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.606703 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 411) took 1503 ms. Trace:
I20250625 13:52:00.606863 1167 rpcz_store.cc:276] 0625 13:51:59.103054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.103111 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:00.519158 (+1416047us) ts_tablet_manager.cc:1145] Deleting tablet 600d711975a74f66b9144758ad0dcfb0
0625 13:52:00.519174 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.585348 (+ 66174us) spinlock_profiling.cc:243] Waited 62.4 ms on lock 0x616000103054. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf7f000 00007f18fd291093 00007f18ffd6e137 00007f18ffda01c4 00007f18ffd81c20 00007f18ffd819ad 0000000000a7573c 00007f18f05649df 00007f18f0567cab 00007f18f0567b28
0625 13:52:00.601821 (+ 16473us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.605413 (+ 3592us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.606674 (+ 1261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":143469312,"tablet-delete.queue_time_us":1415794}
I20250625 13:52:00.610816 1373 raft_consensus.cc:491] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.611006 1373 raft_consensus.cc:513] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.612227 1373 leader_election.cc:290] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:00.612377 1373 raft_consensus.cc:491] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.613632 1373 raft_consensus.cc:513] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.613830 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e6214b878ce4ee6b6152eda7b37e76c" } modify_external_catalogs: true
I20250625 13:52:00.614042 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e6214b878ce4ee6b6152eda7b37e76c" } modify_external_catalogs: true
I20250625 13:52:00.614461 1373 leader_election.cc:290] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:00.624341 1165 raft_consensus.cc:2241] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.625303 1165 raft_consensus.cc:2270] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.626055 756 tablet_service.cc:1515] Processing DeleteTablet for tablet afa1c198dd6541809843244c7cc18194 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.626259 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c605d5b64c164c1594b0a40cfa2c4cf6
W20250625 13:52:00.626626 849 leader_election.cc:343] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.628731 1165 ts_tablet_manager.cc:1905] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.635197 593 master_service.cc:948] ReplaceTablet: received request to replace tablet b7641869a0bb463698244ff445d87a1f from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.637383 756 tablet_service.cc:1515] Processing DeleteTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.653888 1620 catalog_manager.cc:1261] Loaded cluster ID: 4fb72e0fa75741a1972554db6ec14e0e
I20250625 13:52:00.653862 588 master_service.cc:948] ReplaceTablet: received request to replace tablet ae4a105e80c1460cbb21e1bdb9b356b1 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.654349 1620 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: loading cluster ID for follower catalog manager: success
I20250625 13:52:00.655802 1166 tablet_bootstrap.cc:492] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:00.656061 1166 ts_tablet_manager.cc:1397] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.485s user 0.011s sys 0.002s
I20250625 13:52:00.656949 593 master_service.cc:948] ReplaceTablet: received request to replace tablet d85cdadfd694486590092a7b6f61f44c from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.657018 1166 raft_consensus.cc:357] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.657351 1166 raft_consensus.cc:383] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.657466 1166 raft_consensus.cc:738] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.657780 1166 consensus_queue.cc:260] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.658349 1166 ts_tablet_manager.cc:1428] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250625 13:52:00.658638 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 409953us
I20250625 13:52:00.658704 1166 tablet_bootstrap.cc:492] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:00.662559 1166 tablet_bootstrap.cc:654] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.614979 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet d4f19ddc51324cc18242c6081f3f7de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.615092 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 87327f74beb845768bbfd5ae6d2bcf88 with delete_type TABLET_DATA_DELETED (replaced by tablet f00e72e103f443e886f0df690cff4b6f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.628332 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:00.663811 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c3f0ca6fa5482db7f57143911c838f consensus state has no leader
I20250625 13:52:00.664279 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 412) took 1539 ms. Trace:
I20250625 13:52:00.664395 1167 rpcz_store.cc:276] 0625 13:51:59.124604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.124672 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:00.664136 (+1539464us) ts_tablet_manager.cc:1145] Deleting tablet 34ab663b7eba452c9f9775c0f568010b
0625 13:52:00.664145 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.664239 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539206}
I20250625 13:52:00.664882 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 413) took 1526 ms. Trace:
I20250625 13:52:00.665045 1167 rpcz_store.cc:276] 0625 13:51:59.138071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.138123 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:00.664761 (+1526638us) ts_tablet_manager.cc:1145] Deleting tablet 3859f399a7b84de99658380bd6c24ff5
0625 13:52:00.664769 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.664854 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1525380}
I20250625 13:52:00.665252 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 600d711975a74f66b9144758ad0dcfb0 (table default.Table_186697102c744ac39ec9191fc0f992ea [id=548110492fba410e9612ddd9792eed22]) successfully deleted
I20250625 13:52:00.665505 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 414) took 1479 ms. Trace:
W20250625 13:52:00.665547 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 34ab663b7eba452c9f9775c0f568010b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ab663b7eba452c9f9775c0f568010b already in progress: creating tablet
I20250625 13:52:00.665628 1167 rpcz_store.cc:276] 0625 13:51:59.185690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.185755 (+ 65us) service_pool.cc:225] Handling call
0625 13:52:00.665393 (+1479638us) ts_tablet_manager.cc:1145] Deleting tablet 2de9656958164dc4ae2fb0078b38f6c3
0625 13:52:00.665400 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.665474 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1475208}
W20250625 13:52:00.665788 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3859f399a7b84de99658380bd6c24ff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3859f399a7b84de99658380bd6c24ff5 already in progress: creating tablet
W20250625 13:52:00.665990 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2de9656958164dc4ae2fb0078b38f6c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2de9656958164dc4ae2fb0078b38f6c3 already in progress: creating tablet
I20250625 13:52:00.666033 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 415) took 1467 ms. Trace:
I20250625 13:52:00.666132 1167 rpcz_store.cc:276] 0625 13:51:59.198247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.198336 (+ 89us) service_pool.cc:225] Handling call
0625 13:52:00.665925 (+1467589us) ts_tablet_manager.cc:1145] Deleting tablet 69c44d72516e4883aff636ca65929c11
0625 13:52:00.665933 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.666005 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464535}
I20250625 13:52:00.666863 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 417) took 1378 ms. Trace:
I20250625 13:52:00.659992 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf1664afbab647b3baeed6391196bed9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.660579 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c5bea3b6214fa8af2792c5e9c411d9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.658210 1170 tablet_bootstrap.cc:492] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:00.667622 1170 ts_tablet_manager.cc:1397] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.366s user 0.023s sys 0.062s
W20250625 13:52:00.668421 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 69c44d72516e4883aff636ca65929c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69c44d72516e4883aff636ca65929c11 already in progress: creating tablet
I20250625 13:52:00.668766 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd4f0857f52449e1b26fde09c6c7c9d4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.668402 1170 raft_consensus.cc:357] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.672384 1170 raft_consensus.cc:383] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.672489 1170 raft_consensus.cc:738] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.672745 1170 consensus_queue.cc:260] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.674777 1170 ts_tablet_manager.cc:1428] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:00.675107 1170 tablet_bootstrap.cc:492] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:00.675621 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e11bb2795f394504b85419cbea5cb249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.676231 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e11bb2795f394504b85419cbea5cb249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.676856 1170 tablet_bootstrap.cc:654] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.677776 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_46fca0cfc394474781b9a7d01db63df9" } new_table_name: "default.Table_a3e83f37adb44078818e834e91f4e340" modify_external_catalogs: true
W20250625 13:52:00.662617 711 leader_election.cc:343] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.679642 711 leader_election.cc:304] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:00.680054 1669 raft_consensus.cc:2747] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:00.680898 1168 ts_tablet_manager.cc:1905] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:00.681267 711 leader_election.cc:343] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.681408 711 leader_election.cc:304] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:00.681633 1171 tablet_bootstrap.cc:492] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:00.681649 1669 raft_consensus.cc:2747] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:00.681929 711 consensus_peers.cc:487] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:00.682152 711 leader_election.cc:343] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:00.682320 711 leader_election.cc:343] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:00.684213 711 leader_election.cc:343] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.687619 1166 tablet_bootstrap.cc:492] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:00.687840 1166 ts_tablet_manager.cc:1397] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250625 13:52:00.689013 1166 raft_consensus.cc:357] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.689330 1166 raft_consensus.cc:383] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.689502 1166 raft_consensus.cc:738] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.689878 1166 consensus_queue.cc:260] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.676720 1167 rpcz_store.cc:276] 0625 13:51:59.288510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.289022 (+ 512us) service_pool.cc:225] Handling call
0625 13:52:00.666727 (+1377705us) ts_tablet_manager.cc:1145] Deleting tablet d26820eaba98401da3a8b5348be21fc6
0625 13:52:00.666759 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.666832 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355540}
W20250625 13:52:00.695382 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d26820eaba98401da3a8b5348be21fc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d26820eaba98401da3a8b5348be21fc6 already in progress: creating tablet
I20250625 13:52:00.695469 1166 ts_tablet_manager.cc:1428] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:00.695947 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 420) took 1377 ms. Trace:
I20250625 13:52:00.696087 1167 rpcz_store.cc:276] 0625 13:51:59.318093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.318415 (+ 322us) service_pool.cc:225] Handling call
0625 13:52:00.695754 (+1377339us) ts_tablet_manager.cc:1145] Deleting tablet fd39216b01054ff38ec9175d7a2f87f2
0625 13:52:00.695766 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.695905 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376064}
I20250625 13:52:00.696478 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 425) took 1360 ms. Trace:
I20250625 13:52:00.696586 1167 rpcz_store.cc:276] 0625 13:51:59.336131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.338737 (+ 2606us) service_pool.cc:225] Handling call
0625 13:52:00.696346 (+1357609us) ts_tablet_manager.cc:1145] Deleting tablet ecb9a98e402b4da8a9c2bd8317cef532
0625 13:52:00.696357 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.696446 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357386}
I20250625 13:52:00.696916 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 428) took 1331 ms. Trace:
I20250625 13:52:00.697006 1167 rpcz_store.cc:276] 0625 13:51:59.364964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.365117 (+ 153us) service_pool.cc:225] Handling call
0625 13:52:00.696771 (+1331654us) ts_tablet_manager.cc:1145] Deleting tablet 430c3bed360b4f32855a8adb99076608
0625 13:52:00.696779 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.696887 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1331430}
I20250625 13:52:00.697310 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 431) took 1292 ms. Trace:
I20250625 13:52:00.697407 1167 rpcz_store.cc:276] 0625 13:51:59.406856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.407027 (+ 171us) service_pool.cc:225] Handling call
0625 13:52:00.697203 (+1290176us) ts_tablet_manager.cc:1145] Deleting tablet 6f5780d88dea41e7b855b41f12e96c52
0625 13:52:00.697211 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.697282 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280701}
I20250625 13:52:00.697724 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 436) took 1042 ms. Trace:
I20250625 13:52:00.697846 1167 rpcz_store.cc:276] 0625 13:51:59.655018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.655094 (+ 76us) service_pool.cc:225] Handling call
0625 13:52:00.697612 (+1042518us) ts_tablet_manager.cc:1145] Deleting tablet 41a7862601f04fd7a938f7e5d5a4c0f9
0625 13:52:00.697622 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.697698 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042240}
I20250625 13:52:00.698199 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 435) took 1043 ms. Trace:
I20250625 13:52:00.698310 1167 rpcz_store.cc:276] 0625 13:51:59.654665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.654782 (+ 117us) service_pool.cc:225] Handling call
0625 13:52:00.698076 (+1043294us) ts_tablet_manager.cc:1145] Deleting tablet abd6f9121a9d4b39adde1b880a875e23
0625 13:52:00.698083 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.698152 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038888}
I20250625 13:52:00.698613 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 434) took 1045 ms. Trace:
I20250625 13:52:00.698721 1167 rpcz_store.cc:276] 0625 13:51:59.653452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.653519 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:00.698494 (+1044975us) ts_tablet_manager.cc:1145] Deleting tablet 413afd7b62d24fe8b9eceadd57e29fc8
0625 13:52:00.698502 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.698576 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035024}
W20250625 13:52:00.699620 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet fd39216b01054ff38ec9175d7a2f87f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd39216b01054ff38ec9175d7a2f87f2 already in progress: creating tablet
I20250625 13:52:00.699769 591 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 50786a4c25274ffb8f9910c5dafe3933 (table default.Table_ebb1cd6aa381410a82cac38e7197353c [id=d7cbe338b2554a6e99b07325632aaa33]). Expected version 3 got 0
W20250625 13:52:00.699904 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ecb9a98e402b4da8a9c2bd8317cef532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb9a98e402b4da8a9c2bd8317cef532 already in progress: creating tablet
W20250625 13:52:00.700115 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 430c3bed360b4f32855a8adb99076608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 430c3bed360b4f32855a8adb99076608 already in progress: creating tablet
I20250625 13:52:00.700250 1166 tablet_bootstrap.cc:492] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:00.700330 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6f5780d88dea41e7b855b41f12e96c52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f5780d88dea41e7b855b41f12e96c52 already in progress: creating tablet
I20250625 13:52:00.701159 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 389d81329b8d430a9098b0630612fdcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.700559 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e1a2fad7ba6143feb9d139ab60083e8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.702131 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1a2fad7ba6143feb9d139ab60083e8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:00.700575 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41a7862601f04fd7a938f7e5d5a4c0f9 already in progress: creating tablet
W20250625 13:52:00.704186 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet abd6f9121a9d4b39adde1b880a875e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abd6f9121a9d4b39adde1b880a875e23 already in progress: creating tablet
W20250625 13:52:00.704537 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 413afd7b62d24fe8b9eceadd57e29fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 413afd7b62d24fe8b9eceadd57e29fc8 already in progress: creating tablet
I20250625 13:52:00.705029 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 437) took 1041 ms. Trace:
I20250625 13:52:00.709529 1167 rpcz_store.cc:276] 0625 13:51:59.664043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.664088 (+ 45us) service_pool.cc:225] Handling call
0625 13:52:00.698941 (+1034853us) ts_tablet_manager.cc:1145] Deleting tablet 77c50e1ebb9042ef8f0a651ba7186b74
0625 13:52:00.704876 (+ 5935us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.704991 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029397}
I20250625 13:52:00.709928 1166 tablet_bootstrap.cc:654] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:00.710273 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 77c50e1ebb9042ef8f0a651ba7186b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77c50e1ebb9042ef8f0a651ba7186b74 already in progress: creating tablet
I20250625 13:52:00.710479 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 433) took 1086 ms. Trace:
I20250625 13:52:00.710592 1167 rpcz_store.cc:276] 0625 13:51:59.624567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.624702 (+ 135us) service_pool.cc:225] Handling call
0625 13:52:00.710315 (+1085613us) ts_tablet_manager.cc:1145] Deleting tablet cf1664afbab647b3baeed6391196bed9
0625 13:52:00.710326 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.710444 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020651}
W20250625 13:52:00.711688 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet cf1664afbab647b3baeed6391196bed9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf1664afbab647b3baeed6391196bed9 already in progress: creating tablet
W20250625 13:52:00.711931 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4c9efc7631df4ad29112a41f229ae2e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c9efc7631df4ad29112a41f229ae2e9 already in progress: creating tablet
W20250625 13:52:00.712393 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bb648c45ea344fb3bac765c9e06db2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb648c45ea344fb3bac765c9e06db2b1 already in progress: creating tablet
I20250625 13:52:00.712599 1167 tablet_replica.cc:331] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.713013 1167 raft_consensus.cc:2241] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.713465 1167 raft_consensus.cc:2270] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.714993 1167 ts_tablet_manager.cc:1905] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.717239 1646 raft_consensus.cc:491] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.717660 1171 ts_tablet_manager.cc:1397] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.331s user 0.004s sys 0.005s
W20250625 13:52:00.718787 851 leader_election.cc:343] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d9bb7ebc40a14ecaa0c692a8e1117db3
I20250625 13:52:00.719358 886 tablet_service.cc:1515] Processing DeleteTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.719909 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 799f2edd960d45c1ace9195283a042e2 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c3f0ca6fa5482db7f57143911c838f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:00.720141 1646 raft_consensus.cc:513] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.720432 1669 consensus_queue.cc:1035] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:00.720975 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c5bea3b6214fa8af2792c5e9c411d9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:00.721278 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf1664afbab647b3baeed6391196bed9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:00.721505 711 leader_election.cc:343] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.721704 711 leader_election.cc:304] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:00.721992 1669 raft_consensus.cc:2747] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:00.722848 1646 leader_election.cc:290] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:00.723939 711 leader_election.cc:343] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.724138 711 leader_election.cc:304] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:00.724429 1669 raft_consensus.cc:2747] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:00.723899 1171 raft_consensus.cc:357] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.725740 1171 raft_consensus.cc:383] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.725847 1171 raft_consensus.cc:738] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.726058 1171 consensus_queue.cc:260] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.726526 1171 ts_tablet_manager.cc:1428] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250625 13:52:00.726843 1171 tablet_bootstrap.cc:492] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:00.728582 1171 tablet_bootstrap.cc:654] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.729769 1167 ts_tablet_manager.cc:1918] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.729768 851 leader_election.cc:304] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:00.730209 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 116754us
I20250625 13:52:00.730993 1442 raft_consensus.cc:2747] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:00.731547 851 leader_election.cc:343] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.732374 1166 tablet_bootstrap.cc:492] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:00.732578 1166 ts_tablet_manager.cc:1397] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
W20250625 13:52:00.732270 849 leader_election.cc:343] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: cbaa018753d8455199f3869323fb5582
I20250625 13:52:00.733436 1166 raft_consensus.cc:357] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.733660 1166 raft_consensus.cc:383] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.733734 1166 raft_consensus.cc:738] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.734079 1166 consensus_queue.cc:260] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.734356 851 leader_election.cc:304] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:00.734665 1166 ts_tablet_manager.cc:1428] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:00.735292 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd4f0857f52449e1b26fde09c6c7c9d4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:00.735631 1646 raft_consensus.cc:2747] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:00.736912 1167 log.cc:1199] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/14a12a5453e647b9946ca19e0ffb14ee
I20250625 13:52:00.737541 1167 ts_tablet_manager.cc:1939] T 14a12a5453e647b9946ca19e0ffb14ee P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:00.737506 711 leader_election.cc:343] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.737648 711 leader_election.cc:304] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:00.738559 1669 raft_consensus.cc:2747] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:00.738734 1166 tablet_bootstrap.cc:492] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:00.740778 1166 tablet_bootstrap.cc:654] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:00.741181 851 leader_election.cc:343] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cbaa018753d8455199f3869323fb5582
I20250625 13:52:00.741338 851 leader_election.cc:304] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:00.741686 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 14a12a5453e647b9946ca19e0ffb14ee (table default.Table_6cfe68ee62114f35be36b7f12d451a24 [id=09a0b05c11214fd0952325f08a002b8b]) successfully deleted
W20250625 13:52:00.744499 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet dbd2120b4b084691b6604305773e74f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbd2120b4b084691b6604305773e74f9 already in progress: creating tablet
I20250625 13:52:00.745147 1646 raft_consensus.cc:2747] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:00.745618 1167 tablet_replica.cc:331] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.745880 1669 raft_consensus.cc:491] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.746012 1669 raft_consensus.cc:513] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.746706 1669 leader_election.cc:290] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:00.747071 1167 raft_consensus.cc:2241] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.747418 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "643928ca4fc94ce58431a0525f2abb99" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:00.747607 1167 raft_consensus.cc:2270] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:00.747916 711 leader_election.cc:343] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.748170 1167 ts_tablet_manager.cc:1905] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.749974 1689 tablet.cc:1722] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:00.752565 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "643928ca4fc94ce58431a0525f2abb99" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:00.753454 711 leader_election.cc:343] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.753593 711 leader_election.cc:304] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:00.753872 1669 raft_consensus.cc:2747] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:00.754575 1166 tablet_bootstrap.cc:492] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:00.754757 1166 ts_tablet_manager.cc:1397] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20250625 13:52:00.755677 1166 raft_consensus.cc:357] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.755919 1166 raft_consensus.cc:383] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.756044 1166 raft_consensus.cc:738] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.756348 1166 consensus_queue.cc:260] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.757057 1166 ts_tablet_manager.cc:1428] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:00.758248 1166 tablet_bootstrap.cc:492] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:00.695001 1165 ts_tablet_manager.cc:1918] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.762622 1167 ts_tablet_manager.cc:1918] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.763936 1167 log.cc:1199] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/8149aea0ff7646d3b94c9159f56e48ef
I20250625 13:52:00.764678 1167 ts_tablet_manager.cc:1939] T 8149aea0ff7646d3b94c9159f56e48ef P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.766320 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 8149aea0ff7646d3b94c9159f56e48ef (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]) successfully deleted
I20250625 13:52:00.763199 1165 log.cc:1199] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/799f2edd960d45c1ace9195283a042e2
I20250625 13:52:00.767292 1165 ts_tablet_manager.cc:1939] T 799f2edd960d45c1ace9195283a042e2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:00.767539 1167 tablet_replica.cc:331] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.767992 1167 raft_consensus.cc:2241] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:00.768237 1168 ts_tablet_manager.cc:1918] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.768373 1168 log.cc:1199] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/07e281786e6e4a5d93b94b85b486655d
I20250625 13:52:00.768669 1167 pending_rounds.cc:70] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:00.768849 1168 ts_tablet_manager.cc:1939] T 07e281786e6e4a5d93b94b85b486655d P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:00.769060 1167 pending_rounds.cc:77] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520595431698432 op_type: NO_OP noop_request { }
I20250625 13:52:00.769361 1167 raft_consensus.cc:2887] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:00.769626 1167 raft_consensus.cc:2270] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.770766 1167 ts_tablet_manager.cc:1905] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.772691 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 415) took 1920 ms. Trace:
I20250625 13:52:00.772883 1165 rpcz_store.cc:276] 0625 13:51:58.851735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.854000 (+ 2265us) service_pool.cc:225] Handling call
0625 13:52:00.413602 (+1559602us) ts_tablet_manager.cc:1145] Deleting tablet 799f2edd960d45c1ace9195283a042e2
0625 13:52:00.413610 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.672080 (+258470us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.694866 (+ 22786us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.772641 (+ 77775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541461}
I20250625 13:52:00.775254 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 799f2edd960d45c1ace9195283a042e2 (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]) successfully deleted
I20250625 13:52:00.775893 1165 tablet_replica.cc:331] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:00.776381 1165 raft_consensus.cc:2241] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.776870 1166 tablet_bootstrap.cc:654] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.778488 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 643928ca4fc94ce58431a0525f2abb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.778803 753 tablet_service.cc:1515] Processing DeleteTablet for tablet bb1c7a85da844d34be805843bc60d350 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:00.779170 1165 raft_consensus.cc:2270] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.779718 1165 ts_tablet_manager.cc:1905] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.784263 1167 ts_tablet_manager.cc:1918] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.786840 1165 ts_tablet_manager.cc:1918] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.788859 1167 log.cc:1199] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/36da1da539e24f55978d516873cb0b27
I20250625 13:52:00.790035 1167 ts_tablet_manager.cc:1939] T 36da1da539e24f55978d516873cb0b27 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.791879 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 441) took 1017 ms. Trace:
I20250625 13:52:00.792951 1167 rpcz_store.cc:276] 0625 13:51:59.773880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.773931 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:00.767478 (+993547us) ts_tablet_manager.cc:1145] Deleting tablet 36da1da539e24f55978d516873cb0b27
0625 13:52:00.767487 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.781070 (+ 13583us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.784111 (+ 3041us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.791815 (+ 7704us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":903432}
I20250625 13:52:00.779278 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 87f5b36ddb474ed6903c14876b236ba8 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.793797 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 36da1da539e24f55978d516873cb0b27 (table default.Table_b079a0f225cf4ac98c2efc03bb74b889 [id=83a712e3adf74a95a5c2ea3fc5c0e14e]) successfully deleted
I20250625 13:52:00.794370 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 443) took 1018 ms. Trace:
I20250625 13:52:00.794870 1167 rpcz_store.cc:276] 0625 13:51:59.775806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.775855 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:00.794195 (+1018340us) ts_tablet_manager.cc:1145] Deleting tablet 7d361e73fe3d4ea2bb6735f814345ebb
0625 13:52:00.794206 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.794332 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926800}
W20250625 13:52:00.795619 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7d361e73fe3d4ea2bb6735f814345ebb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d361e73fe3d4ea2bb6735f814345ebb already in progress: creating tablet
I20250625 13:52:00.796212 1669 raft_consensus.cc:491] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:00.796352 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet dcf7fab53cae49908d2952cdaa07943a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcf7fab53cae49908d2952cdaa07943a already in progress: creating tablet
I20250625 13:52:00.796393 1669 raft_consensus.cc:513] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.797221 1669 leader_election.cc:290] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:00.797236 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 077e8c001cc646559b5268d4a48c0ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 077e8c001cc646559b5268d4a48c0ad6 already in progress: creating tablet
W20250625 13:52:00.798079 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7efc8df3c8e24020b073a1b61bc9e0a0 already in progress: creating tablet
I20250625 13:52:00.798059 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e58fbec94e4ff9b7ac9830b576fe7f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:00.798669 711 leader_election.cc:343] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:00.798988 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 829221b8a79f460d90dd24be6a2d4c43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829221b8a79f460d90dd24be6a2d4c43 already in progress: creating tablet
W20250625 13:52:00.799886 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 150af49427ae47b98e9b2077bbbe1e2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 150af49427ae47b98e9b2077bbbe1e2b already in progress: creating tablet
W20250625 13:52:00.800751 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ff65e23c44624b2cbf92c2dc67eaed09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff65e23c44624b2cbf92c2dc67eaed09 already in progress: creating tablet
W20250625 13:52:00.801543 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ed19663d21d84c9e90d4f4dda055420d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed19663d21d84c9e90d4f4dda055420d already in progress: creating tablet
W20250625 13:52:00.802328 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet cbf7e30cdff245208dacc6963fe88c7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbf7e30cdff245208dacc6963fe88c7d already in progress: creating tablet
W20250625 13:52:00.803117 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet efc9536a716f4b9f9fc4af1637b3cd80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efc9536a716f4b9f9fc4af1637b3cd80 already in progress: creating tablet
W20250625 13:52:00.803953 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet afa1c198dd6541809843244c7cc18194 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afa1c198dd6541809843244c7cc18194 already in progress: creating tablet
W20250625 13:52:00.804733 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d4f19ddc51324cc18242c6081f3f7de5 because the tablet was not found. No further retry: Not found: Tablet not found: d4f19ddc51324cc18242c6081f3f7de5
I20250625 13:52:00.805305 1167 tablet_replica.cc:331] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.806138 1167 raft_consensus.cc:2241] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.806720 1167 raft_consensus.cc:2270] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.807608 1167 ts_tablet_manager.cc:1905] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.808580 1165 log.cc:1199] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/85c72f9ae92a4537979fb479eb4467c4
I20250625 13:52:00.809048 1165 ts_tablet_manager.cc:1939] T 85c72f9ae92a4537979fb479eb4467c4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:00.810497 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 414) took 2003 ms. Trace:
I20250625 13:52:00.811506 1170 tablet_bootstrap.cc:492] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:00.811726 1170 ts_tablet_manager.cc:1397] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.137s user 0.012s sys 0.032s
I20250625 13:52:00.811551 1165 rpcz_store.cc:276] 0625 13:51:58.807007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.811434 (+ 4427us) service_pool.cc:225] Handling call
0625 13:52:00.775797 (+1964363us) ts_tablet_manager.cc:1145] Deleting tablet 85c72f9ae92a4537979fb479eb4467c4
0625 13:52:00.775808 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.783796 (+ 7988us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.786709 (+ 2913us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.810456 (+ 23747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1895131}
I20250625 13:52:00.812343 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 409) took 2005 ms. Trace:
I20250625 13:52:00.812680 1170 raft_consensus.cc:357] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.812944 1170 raft_consensus.cc:383] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.813057 1170 raft_consensus.cc:738] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.813061 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e58fbec94e4ff9b7ac9830b576fe7f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:00.812916 1165 rpcz_store.cc:276] 0625 13:51:58.806479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.807052 (+ 573us) service_pool.cc:225] Handling call
0625 13:52:00.812180 (+2005128us) ts_tablet_manager.cc:1145] Deleting tablet 38a23b08a03b48eaa051f670eb7dac65
0625 13:52:00.812191 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.812311 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1931368}
I20250625 13:52:00.813416 1165 tablet_replica.cc:331] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:00.813360 1170 consensus_queue.cc:260] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.813620 1165 raft_consensus.cc:2241] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:00.813841 1170 ts_tablet_manager.cc:1428] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:00.813895 1165 pending_rounds.cc:70] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:52:00.813972 1165 pending_rounds.cc:77] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520592048504832 op_type: NO_OP noop_request { }
I20250625 13:52:00.814062 1165 raft_consensus.cc:2887] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:00.814121 1170 tablet_bootstrap.cc:492] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:00.814167 1165 raft_consensus.cc:2270] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.814666 1165 ts_tablet_manager.cc:1905] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.815078 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 85c72f9ae92a4537979fb479eb4467c4 (table default.Table_8fc2c990732d4e658580c2bc2cf67f4d [id=75b34bf006e040d18266d0c90986afeb]) successfully deleted
W20250625 13:52:00.815364 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 38a23b08a03b48eaa051f670eb7dac65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38a23b08a03b48eaa051f670eb7dac65 already in progress: creating tablet
W20250625 13:52:00.815773 711 leader_election.cc:343] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:00.815960 711 leader_election.cc:304] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:00.816160 1170 tablet_bootstrap.cc:654] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.816495 1669 raft_consensus.cc:2747] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:00.769187 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 201) took 5398 ms. Trace:
I20250625 13:52:00.817795 1168 rpcz_store.cc:276] 0625 13:51:55.370483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.370667 (+ 184us) service_pool.cc:225] Handling call
0625 13:52:00.335752 (+4965085us) ts_tablet_manager.cc:1145] Deleting tablet 07e281786e6e4a5d93b94b85b486655d
0625 13:52:00.335765 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.546225 (+210460us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x616000093154. stack: 00007f37024dc336 00007f37024dc0f0 00007f37014d23ad 0000000000971004 0000000000970fcd 000000000097099a 00007f370df3c000 00007f370f24e093 00007f3711d2b137 00007f3711d5d1c4 00007f3711d3ec20 00007f3711d3e9ad 0000000000a7573c 00007f37025219df 00007f3702524cab 00007f3702524b28
0625 13:52:00.732416 (+186191us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.768064 (+ 35648us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.769155 (+ 1091us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":482670976,"tablet-delete.queue_time_us":4964864}
I20250625 13:52:00.818336 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 07e281786e6e4a5d93b94b85b486655d (table default.Table_fb311c42df024421b325596e0cb7a778 [id=7b67603755e7428496524ee716b435e9]) successfully deleted
I20250625 13:52:00.818838 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 202) took 5448 ms. Trace:
I20250625 13:52:00.819002 1168 rpcz_store.cc:276] 0625 13:51:55.370617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.370915 (+ 298us) service_pool.cc:225] Handling call
0625 13:52:00.818673 (+5447758us) ts_tablet_manager.cc:1145] Deleting tablet 6771d8e180fe48569a0e016db0cc9b51
0625 13:52:00.818683 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.818805 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5447621}
W20250625 13:52:00.819465 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6771d8e180fe48569a0e016db0cc9b51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6771d8e180fe48569a0e016db0cc9b51 already in progress: creating tablet
I20250625 13:52:00.820108 1168 tablet_replica.cc:331] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:00.820361 1168 raft_consensus.cc:2241] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:00.825016 1167 ts_tablet_manager.cc:1918] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.826969 1167 log.cc:1199] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/87327f74beb845768bbfd5ae6d2bcf88
I20250625 13:52:00.827944 1167 ts_tablet_manager.cc:1939] T 87327f74beb845768bbfd5ae6d2bcf88 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:00.828336 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d139478986e346e2ac833ce353cff7e1 has no consensus state
I20250625 13:52:00.829734 590 master_service.cc:948] ReplaceTablet: received request to replace tablet cd4f0857f52449e1b26fde09c6c7c9d4 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:00.830080 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1fc704a30db04ed98f279746d4f875b2" } new_table_name: "default.Table_0c326adff3bb4c6ea214f62d5a9c3e86" modify_external_catalogs: true
I20250625 13:52:00.830417 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_20227b2471d345dc90782795a2d1de32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.831120 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20227b2471d345dc90782795a2d1de32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.831368 1167 tablet_replica.cc:331] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.832037 1170 tablet_bootstrap.cc:492] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:00.829768 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 87327f74beb845768bbfd5ae6d2bcf88 (table default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]) successfully deleted
I20250625 13:52:00.832311 1170 ts_tablet_manager.cc:1397] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
W20250625 13:52:00.832525 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c605d5b64c164c1594b0a40cfa2c4cf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c605d5b64c164c1594b0a40cfa2c4cf6 already in progress: creating tablet
I20250625 13:52:00.833246 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5ba76c130a4711af5b21e9eb8562e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:00.833365 1170 raft_consensus.cc:357] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.833688 1170 raft_consensus.cc:383] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.833825 1170 raft_consensus.cc:738] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.834134 1170 consensus_queue.cc:260] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.834233 1168 pending_rounds.cc:70] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:00.835506 1170 ts_tablet_manager.cc:1428] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:00.835765 1167 raft_consensus.cc:2241] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.836011 1167 raft_consensus.cc:2270] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.836838 1167 ts_tablet_manager.cc:1905] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.837414 1687 tablet.cc:1722] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:00.838362 1168 pending_rounds.cc:77] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520595838795776 op_type: NO_OP noop_request { }
I20250625 13:52:00.838512 1168 raft_consensus.cc:2887] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:00.838656 1168 raft_consensus.cc:2270] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.840998 1165 ts_tablet_manager.cc:1918] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:00.842531 1165 log.cc:1199] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/583ee2f2c66841939c93104efc36facb
I20250625 13:52:00.843223 1165 ts_tablet_manager.cc:1939] T 583ee2f2c66841939c93104efc36facb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:00.844606 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 417) took 1924 ms. Trace:
I20250625 13:52:00.845633 1165 rpcz_store.cc:276] 0625 13:51:58.920579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.922344 (+ 1765us) service_pool.cc:225] Handling call
0625 13:52:00.813346 (+1891002us) ts_tablet_manager.cc:1145] Deleting tablet 583ee2f2c66841939c93104efc36facb
0625 13:52:00.813355 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.817949 (+ 4594us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.840819 (+ 22870us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.844562 (+ 3743us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867006}
I20250625 13:52:00.846359 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 418) took 1899 ms. Trace:
I20250625 13:52:00.846426 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 583ee2f2c66841939c93104efc36facb (table default.Table_5964baca5b4a4701bf5a1b86d494b4e0 [id=c12c680f4ece4270ba65d0e2dc275254]) successfully deleted
I20250625 13:52:00.846796 1165 rpcz_store.cc:276] 0625 13:51:58.947221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.947604 (+ 383us) service_pool.cc:225] Handling call
0625 13:52:00.846214 (+1898610us) ts_tablet_manager.cc:1145] Deleting tablet ccce48011a15405db2b95c04c9b01b3c
0625 13:52:00.846222 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.846309 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878222}
I20250625 13:52:00.847081 1165 tablet_replica.cc:331] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:52:00.847361 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ccce48011a15405db2b95c04c9b01b3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccce48011a15405db2b95c04c9b01b3c already in progress: creating tablet
I20250625 13:52:00.847537 1170 tablet_bootstrap.cc:492] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:00.847666 1165 raft_consensus.cc:2241] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:00.848030 1165 raft_consensus.cc:2270] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.848605 1165 ts_tablet_manager.cc:1905] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.849984 1170 tablet_bootstrap.cc:654] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.854470 1604 raft_consensus.cc:491] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.854648 1604 raft_consensus.cc:513] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.855422 1604 leader_election.cc:290] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:00.857056 851 leader_election.cc:343] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1a7b72c2c464421eacfcf64008c3aea8
W20250625 13:52:00.857730 849 leader_election.cc:343] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 1a7b72c2c464421eacfcf64008c3aea8
I20250625 13:52:00.857892 849 leader_election.cc:304] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:00.858462 1604 raft_consensus.cc:2747] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:00.862370 1165 ts_tablet_manager.cc:1918] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:00.864203 1165 log.cc:1199] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d4f19ddc51324cc18242c6081f3f7de5
I20250625 13:52:00.864826 1165 ts_tablet_manager.cc:1939] T d4f19ddc51324cc18242c6081f3f7de5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:00.866133 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 419) took 1842 ms. Trace:
I20250625 13:52:00.867167 1165 rpcz_store.cc:276] 0625 13:51:59.023252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.023800 (+ 548us) service_pool.cc:225] Handling call
0625 13:52:00.847023 (+1823223us) ts_tablet_manager.cc:1145] Deleting tablet d4f19ddc51324cc18242c6081f3f7de5
0625 13:52:00.847030 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.855899 (+ 8869us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.862245 (+ 6346us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.866073 (+ 3828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800827}
W20250625 13:52:00.867951 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93003ef3621a4f33acbc7eff54a796e3 has no consensus state
I20250625 13:52:00.869480 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d4f19ddc51324cc18242c6081f3f7de5 (table default.Table_08c36a3d2a03417982751548b18def03 [id=b8285b64bbee49f9be8741b252d03363]) successfully deleted
I20250625 13:52:00.874163 1170 tablet_bootstrap.cc:492] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:00.874281 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 420) took 1742 ms. Trace:
I20250625 13:52:00.874493 1170 ts_tablet_manager.cc:1397] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250625 13:52:00.874428 1165 rpcz_store.cc:276] 0625 13:51:59.125529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.125598 (+ 69us) service_pool.cc:225] Handling call
0625 13:52:00.867891 (+1742293us) ts_tablet_manager.cc:1145] Deleting tablet 34ab663b7eba452c9f9775c0f568010b
0625 13:52:00.867902 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.868340 (+ 438us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742060}
I20250625 13:52:00.875558 1170 raft_consensus.cc:357] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.875664 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 421) took 1736 ms. Trace:
W20250625 13:52:00.875701 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 34ab663b7eba452c9f9775c0f568010b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ab663b7eba452c9f9775c0f568010b already in progress: creating tablet
I20250625 13:52:00.875757 1165 rpcz_store.cc:276] 0625 13:51:59.138891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.138941 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:00.875463 (+1736522us) ts_tablet_manager.cc:1145] Deleting tablet 3859f399a7b84de99658380bd6c24ff5
0625 13:52:00.875473 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.875611 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1736292}
I20250625 13:52:00.876170 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 422) took 1698 ms. Trace:
W20250625 13:52:00.876206 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3859f399a7b84de99658380bd6c24ff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3859f399a7b84de99658380bd6c24ff5 already in progress: creating tablet
I20250625 13:52:00.876276 1165 rpcz_store.cc:276] 0625 13:51:59.177311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.177367 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:00.876017 (+1698650us) ts_tablet_manager.cc:1145] Deleting tablet 2de9656958164dc4ae2fb0078b38f6c3
0625 13:52:00.876027 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.876121 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698409}
I20250625 13:52:00.876430 1170 raft_consensus.cc:383] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:00.876582 1170 raft_consensus.cc:738] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:00.876647 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 423) took 1677 ms. Trace:
W20250625 13:52:00.876772 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2de9656958164dc4ae2fb0078b38f6c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2de9656958164dc4ae2fb0078b38f6c3 already in progress: creating tablet
I20250625 13:52:00.876747 1165 rpcz_store.cc:276] 0625 13:51:59.198992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.199053 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:00.876482 (+1677429us) ts_tablet_manager.cc:1145] Deleting tablet 69c44d72516e4883aff636ca65929c11
0625 13:52:00.876490 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.876590 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1677183}
I20250625 13:52:00.877058 1165 tablet_replica.cc:331] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:00.877097 1170 consensus_queue.cc:260] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:00.877255 1165 raft_consensus.cc:2241] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:00.877283 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 69c44d72516e4883aff636ca65929c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69c44d72516e4883aff636ca65929c11 already in progress: creating tablet
I20250625 13:52:00.877454 1165 raft_consensus.cc:2270] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:00.877630 851 consensus_peers.cc:487] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 600d711975a74f66b9144758ad0dcfb0. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:00.878038 1165 ts_tablet_manager.cc:1905] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.878827 1620 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: acquiring CA information for follower catalog manager: success
I20250625 13:52:00.881422 1170 ts_tablet_manager.cc:1428] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 13:52:00.883997 1170 tablet_bootstrap.cc:492] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:00.886359 1170 tablet_bootstrap.cc:654] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:00.884671 1167 ts_tablet_manager.cc:1918] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.887439 1167 log.cc:1199] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/389d81329b8d430a9098b0630612fdcb
I20250625 13:52:00.887913 1167 ts_tablet_manager.cc:1939] T 389d81329b8d430a9098b0630612fdcb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.888919 1167 tablet_replica.cc:331] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:00.889112 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 389d81329b8d430a9098b0630612fdcb (table default.Table_1a06c7c260544cce91ab86dad2d62346 [id=aff513ce6735496784852aef3728d643]) successfully deleted
I20250625 13:52:00.889174 1167 raft_consensus.cc:2241] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.889328 1167 raft_consensus.cc:2270] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:00.890097 1167 ts_tablet_manager.cc:1905] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.890480 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5f921e5e270c4b868cc0c36c900919d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:00.891072 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f921e5e270c4b868cc0c36c900919d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:00.892948 1168 ts_tablet_manager.cc:1905] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:00.893858 1165 ts_tablet_manager.cc:1918] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.894485 1165 log.cc:1199] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/1f6be7189f8f45ccb329ef5d2fd8da8d
I20250625 13:52:00.895148 1165 ts_tablet_manager.cc:1939] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:00.897696 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 425) took 1678 ms. Trace:
I20250625 13:52:00.897776 1167 ts_tablet_manager.cc:1918] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:00.898013 1165 rpcz_store.cc:276] 0625 13:51:59.219634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.219691 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:00.876974 (+1657283us) ts_tablet_manager.cc:1145] Deleting tablet 1f6be7189f8f45ccb329ef5d2fd8da8d
0625 13:52:00.876981 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.884251 (+ 7270us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.893699 (+ 9448us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:00.897653 (+ 3954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1656235}
I20250625 13:52:00.898833 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 1f6be7189f8f45ccb329ef5d2fd8da8d (table default.Table_1be3d9db61ab4b56b98b70af09df98b3 [id=68e44a6915224107b42ca15194e2fed9]) successfully deleted
I20250625 13:52:00.899474 1165 tablet_replica.cc:331] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:00.900071 1165 raft_consensus.cc:2241] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:00.902377 1167 log.cc:1199] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/bd5ba76c130a4711af5b21e9eb8562e7
I20250625 13:52:00.902998 1167 ts_tablet_manager.cc:1939] T bd5ba76c130a4711af5b21e9eb8562e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:00.904955 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 93fdde7d068b400ba7070166eff63502 (DEFAULT_TABLE table=default.Table_0089084e327b47a1a45bf11be8539647 [id=fbd907a57d874bb7b319ecd03f6025a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.905516 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fdde7d068b400ba7070166eff63502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.907714 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 (DEFAULT_TABLE table=default.Table_4cf923b8b24141eeab3c245e08664637 [id=62547e04cd8e40bb89291a1462ce8e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.908201 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4b428ae7bc49fcbc37e9d7d238ec21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.908937 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet bd5ba76c130a4711af5b21e9eb8562e7 (table default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]) successfully deleted
I20250625 13:52:00.909677 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 58a532e889244b088562081e444234a3 (DEFAULT_TABLE table=default.Table_3f882e0fab474cf2aef02b0067d704f8 [id=515860fa91ae43bea4530116b0306433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.910018 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58a532e889244b088562081e444234a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.914532 887 tablet_service.cc:1468] Processing CreateTablet for tablet 93fdde7d068b400ba7070166eff63502 (DEFAULT_TABLE table=default.Table_0089084e327b47a1a45bf11be8539647 [id=fbd907a57d874bb7b319ecd03f6025a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.914569 893 tablet_service.cc:1468] Processing CreateTablet for tablet 54cc2d0cf95b41d6b36483af5436fcde (DEFAULT_TABLE table=default.Table_1653a8a7a93d4e77abf24438181cb8ec [id=3a99fd57f313483096bddfdb698c0ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.914973 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fdde7d068b400ba7070166eff63502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.915146 894 tablet_service.cc:1468] Processing CreateTablet for tablet c737daaede054d09a0c07564157423a9 (DEFAULT_TABLE table=default.Table_ddd89f53a2204d2f879f8fd45e20a888 [id=ce3e680405a44e1bb7eea5c4d87bc702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.915743 883 tablet_service.cc:1468] Processing CreateTablet for tablet 58a532e889244b088562081e444234a3 (DEFAULT_TABLE table=default.Table_3f882e0fab474cf2aef02b0067d704f8 [id=515860fa91ae43bea4530116b0306433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.919654 886 tablet_service.cc:1468] Processing CreateTablet for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 (DEFAULT_TABLE table=default.Table_4cf923b8b24141eeab3c245e08664637 [id=62547e04cd8e40bb89291a1462ce8e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.920272 878 tablet_service.cc:1468] Processing CreateTablet for tablet d139478986e346e2ac833ce353cff7e1 (DEFAULT_TABLE table=default.Table_f766c58c1858460cbad7a7f98f1ac55e [id=de29329fd5674a0ab6de40073b58dff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.921304 888 tablet_service.cc:1468] Processing CreateTablet for tablet 254afffd283449fdaf5e190ba3028f16 (DEFAULT_TABLE table=default.Table_865a0d33634943c88b8a9b87a6cae4b4 [id=98ea4f7889a1437b8fcd92c566ca9e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.922058 890 tablet_service.cc:1468] Processing CreateTablet for tablet f0a59da2129141a698692c4b31609d1d (DEFAULT_TABLE table=default.Table_0220bc5deb894b909ecd95a1724cf43d [id=19f6618ce1cd4449a6ba04413f1efd6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.922478 892 tablet_service.cc:1468] Processing CreateTablet for tablet 944759869ced45f09dfad8959b32edff (DEFAULT_TABLE table=default.Table_af5d40e690b54d16b4689c880fc20261 [id=35cb8419cfc5485e9da0e1e37d1f1454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.911583 756 tablet_service.cc:1468] Processing CreateTablet for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 (DEFAULT_TABLE table=default.Table_4cf923b8b24141eeab3c245e08664637 [id=62547e04cd8e40bb89291a1462ce8e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.922996 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4b428ae7bc49fcbc37e9d7d238ec21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.922955 891 tablet_service.cc:1468] Processing CreateTablet for tablet e8d2ab8f91a045a3a39f3d09459f2a28 (DEFAULT_TABLE table=default.Table_25010c89df9e4e128b8e4531a0bda6d3 [id=4f839189b02c499c896a5b8561780423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.924109 744 tablet_service.cc:1468] Processing CreateTablet for tablet e1b4da58289342d991cdf278005b96a0 (DEFAULT_TABLE table=default.Table_ff33e931cb1c4e44a12f7dcb8ddad00c [id=f689bfaf5bea47b9a008c2657e3e15eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.924484 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1b4da58289342d991cdf278005b96a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.911926 758 tablet_service.cc:1468] Processing CreateTablet for tablet 54cc2d0cf95b41d6b36483af5436fcde (DEFAULT_TABLE table=default.Table_1653a8a7a93d4e77abf24438181cb8ec [id=3a99fd57f313483096bddfdb698c0ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.928260 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54cc2d0cf95b41d6b36483af5436fcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.929534 1011 tablet_service.cc:1468] Processing CreateTablet for tablet f0a59da2129141a698692c4b31609d1d (DEFAULT_TABLE table=default.Table_0220bc5deb894b909ecd95a1724cf43d [id=19f6618ce1cd4449a6ba04413f1efd6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.930639 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a59da2129141a698692c4b31609d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.912215 748 tablet_service.cc:1468] Processing CreateTablet for tablet 58a532e889244b088562081e444234a3 (DEFAULT_TABLE table=default.Table_3f882e0fab474cf2aef02b0067d704f8 [id=515860fa91ae43bea4530116b0306433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.931402 889 tablet_service.cc:1468] Processing CreateTablet for tablet e99fe5ad690049029b8325dd38a11580 (DEFAULT_TABLE table=default.Table_66186ab6312345c1a94ca39ad501c296 [id=20cbf73d64ee43899a58b19cd4dad71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.931630 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58a532e889244b088562081e444234a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.931725 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 254afffd283449fdaf5e190ba3028f16 (DEFAULT_TABLE table=default.Table_865a0d33634943c88b8a9b87a6cae4b4 [id=98ea4f7889a1437b8fcd92c566ca9e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.927767 882 tablet_service.cc:1468] Processing CreateTablet for tablet c84b93a9079e45bab55353734ebe7cc2 (DEFAULT_TABLE table=default.Table_01f1a91fcc43403db4e9c2092f12a888 [id=80b9adbbcc284923a66a85410f35da22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.932166 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 944759869ced45f09dfad8959b32edff (DEFAULT_TABLE table=default.Table_af5d40e690b54d16b4689c880fc20261 [id=35cb8419cfc5485e9da0e1e37d1f1454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.932401 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 54cc2d0cf95b41d6b36483af5436fcde (DEFAULT_TABLE table=default.Table_1653a8a7a93d4e77abf24438181cb8ec [id=3a99fd57f313483096bddfdb698c0ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.932617 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1f4d6be016284dc796945270178b9ee1 (DEFAULT_TABLE table=default.Table_9bc32b4aab234a4e8bab3910a4a3102b [id=cb4133ffe2c44743bdafdfafb7899843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.912539 751 tablet_service.cc:1468] Processing CreateTablet for tablet d139478986e346e2ac833ce353cff7e1 (DEFAULT_TABLE table=default.Table_f766c58c1858460cbad7a7f98f1ac55e [id=de29329fd5674a0ab6de40073b58dff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.935590 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d139478986e346e2ac833ce353cff7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.937846 1518 raft_consensus.cc:491] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:00.938585 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254afffd283449fdaf5e190ba3028f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.912875 747 tablet_service.cc:1468] Processing CreateTablet for tablet 254afffd283449fdaf5e190ba3028f16 (DEFAULT_TABLE table=default.Table_865a0d33634943c88b8a9b87a6cae4b4 [id=98ea4f7889a1437b8fcd92c566ca9e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.939705 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254afffd283449fdaf5e190ba3028f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.941900 1518 raft_consensus.cc:513] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:00.942814 1518 leader_election.cc:290] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:00.932327 1020 tablet_service.cc:1468] Processing CreateTablet for tablet d139478986e346e2ac833ce353cff7e1 (DEFAULT_TABLE table=default.Table_f766c58c1858460cbad7a7f98f1ac55e [id=de29329fd5674a0ab6de40073b58dff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.913214 749 tablet_service.cc:1468] Processing CreateTablet for tablet f0a59da2129141a698692c4b31609d1d (DEFAULT_TABLE table=default.Table_0220bc5deb894b909ecd95a1724cf43d [id=19f6618ce1cd4449a6ba04413f1efd6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.943590 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a59da2129141a698692c4b31609d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.943650 1028 tablet_service.cc:1468] Processing CreateTablet for tablet c84b93a9079e45bab55353734ebe7cc2 (DEFAULT_TABLE table=default.Table_01f1a91fcc43403db4e9c2092f12a888 [id=80b9adbbcc284923a66a85410f35da22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.946007 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84b93a9079e45bab55353734ebe7cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.946909 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e8d2ab8f91a045a3a39f3d09459f2a28 (DEFAULT_TABLE table=default.Table_25010c89df9e4e128b8e4531a0bda6d3 [id=4f839189b02c499c896a5b8561780423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.913547 752 tablet_service.cc:1468] Processing CreateTablet for tablet 944759869ced45f09dfad8959b32edff (DEFAULT_TABLE table=default.Table_af5d40e690b54d16b4689c880fc20261 [id=35cb8419cfc5485e9da0e1e37d1f1454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.947589 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 944759869ced45f09dfad8959b32edff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.948264 877 tablet_service.cc:1468] Processing CreateTablet for tablet dc350f8573f74dd88dcbc4c9a7c192c4 (DEFAULT_TABLE table=default.Table_1fc704a30db04ed98f279746d4f875b2 [id=4f2def4377f841b0be3657b4ee82f73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.913859 755 tablet_service.cc:1468] Processing CreateTablet for tablet e8d2ab8f91a045a3a39f3d09459f2a28 (DEFAULT_TABLE table=default.Table_25010c89df9e4e128b8e4531a0bda6d3 [id=4f839189b02c499c896a5b8561780423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.914201 741 tablet_service.cc:1468] Processing CreateTablet for tablet c84b93a9079e45bab55353734ebe7cc2 (DEFAULT_TABLE table=default.Table_01f1a91fcc43403db4e9c2092f12a888 [id=80b9adbbcc284923a66a85410f35da22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.948778 885 tablet_service.cc:1468] Processing CreateTablet for tablet a079e985358040e78455ac8cb6e49f16 (DEFAULT_TABLE table=default.Table_aba4565f0af54d4b984435978a9c2821 [id=1ed1f7751fcd492ca7a928c7a0f491c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.915570 746 tablet_service.cc:1468] Processing CreateTablet for tablet e99fe5ad690049029b8325dd38a11580 (DEFAULT_TABLE table=default.Table_66186ab6312345c1a94ca39ad501c296 [id=20cbf73d64ee43899a58b19cd4dad71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.916157 757 tablet_service.cc:1468] Processing CreateTablet for tablet a079e985358040e78455ac8cb6e49f16 (DEFAULT_TABLE table=default.Table_aba4565f0af54d4b984435978a9c2821 [id=1ed1f7751fcd492ca7a928c7a0f491c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.916311 740 tablet_service.cc:1468] Processing CreateTablet for tablet c737daaede054d09a0c07564157423a9 (DEFAULT_TABLE table=default.Table_ddd89f53a2204d2f879f8fd45e20a888 [id=ce3e680405a44e1bb7eea5c4d87bc702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.949076 896 tablet_service.cc:1468] Processing CreateTablet for tablet 50ac647b1d704142b94da8d506319466 (DEFAULT_TABLE table=default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.911247 753 tablet_service.cc:1468] Processing CreateTablet for tablet 93fdde7d068b400ba7070166eff63502 (DEFAULT_TABLE table=default.Table_0089084e327b47a1a45bf11be8539647 [id=fbd907a57d874bb7b319ecd03f6025a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.917438 745 tablet_service.cc:1468] Processing CreateTablet for tablet dc350f8573f74dd88dcbc4c9a7c192c4 (DEFAULT_TABLE table=default.Table_1fc704a30db04ed98f279746d4f875b2 [id=4f2def4377f841b0be3657b4ee82f73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.918474 750 tablet_service.cc:1468] Processing CreateTablet for tablet 39da3ebbe55f4c4fab4f8af444657b9b (DEFAULT_TABLE table=default.Table_fed63050407b40569f0a8911e9d1fd21 [id=6e495c98f9344b37afc59f947f596b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.919273 743 tablet_service.cc:1468] Processing CreateTablet for tablet 50ac647b1d704142b94da8d506319466 (DEFAULT_TABLE table=default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.920709 754 tablet_service.cc:1468] Processing CreateTablet for tablet 1f4d6be016284dc796945270178b9ee1 (DEFAULT_TABLE table=default.Table_9bc32b4aab234a4e8bab3910a4a3102b [id=cb4133ffe2c44743bdafdfafb7899843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:00.952215 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84b93a9079e45bab55353734ebe7cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.953184 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8d2ab8f91a045a3a39f3d09459f2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.953402 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c737daaede054d09a0c07564157423a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.954221 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39da3ebbe55f4c4fab4f8af444657b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.957664 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc350f8573f74dd88dcbc4c9a7c192c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.958567 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4d6be016284dc796945270178b9ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:00.981943 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99fe5ad690049029b8325dd38a11580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.000571 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fdde7d068b400ba7070166eff63502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.002439 1165 raft_consensus.cc:2270] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.005795 1165 ts_tablet_manager.cc:1905] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.006507 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ac647b1d704142b94da8d506319466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.009884 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a079e985358040e78455ac8cb6e49f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.017555 744 tablet_service.cc:1468] Processing CreateTablet for tablet 93003ef3621a4f33acbc7eff54a796e3 (DEFAULT_TABLE table=default.Table_a3e83f37adb44078818e834e91f4e340 [id=a409c5484b27429f8e269705f3aac00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.019879 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93003ef3621a4f33acbc7eff54a796e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.049655 1021 tablet_service.cc:1468] Processing CreateTablet for tablet c737daaede054d09a0c07564157423a9 (DEFAULT_TABLE table=default.Table_ddd89f53a2204d2f879f8fd45e20a888 [id=ce3e680405a44e1bb7eea5c4d87bc702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.050995 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8d2ab8f91a045a3a39f3d09459f2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.054638 1669 raft_consensus.cc:491] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.054857 1669 raft_consensus.cc:513] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.046815 1620 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 13:52:01.055740 1669 leader_election.cc:290] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.057490 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54cc2d0cf95b41d6b36483af5436fcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.066511 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e99fe5ad690049029b8325dd38a11580 (DEFAULT_TABLE table=default.Table_66186ab6312345c1a94ca39ad501c296 [id=20cbf73d64ee43899a58b19cd4dad71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.067396 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c737daaede054d09a0c07564157423a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.068889 1015 tablet_service.cc:1468] Processing CreateTablet for tablet dc350f8573f74dd88dcbc4c9a7c192c4 (DEFAULT_TABLE table=default.Table_1fc704a30db04ed98f279746d4f875b2 [id=4f2def4377f841b0be3657b4ee82f73d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:01.070194 985 leader_election.cc:343] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d524089e6ab044ec961cfc9f587e8f86
I20250625 13:52:01.070276 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04479dfb34584f0891c604ef72bf93b5" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:01.070365 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 125101us
I20250625 13:52:01.070819 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 39da3ebbe55f4c4fab4f8af444657b9b (DEFAULT_TABLE table=default.Table_fed63050407b40569f0a8911e9d1fd21 [id=6e495c98f9344b37afc59f947f596b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.071627 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a079e985358040e78455ac8cb6e49f16 (DEFAULT_TABLE table=default.Table_aba4565f0af54d4b984435978a9c2821 [id=1ed1f7751fcd492ca7a928c7a0f491c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.072700 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99fe5ad690049029b8325dd38a11580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.075935 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39da3ebbe55f4c4fab4f8af444657b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.077450 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc350f8573f74dd88dcbc4c9a7c192c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:01.082974 711 leader_election.cc:343] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.083415 1026 tablet_service.cc:1468] Processing CreateTablet for tablet e1b4da58289342d991cdf278005b96a0 (DEFAULT_TABLE table=default.Table_ff33e931cb1c4e44a12f7dcb8ddad00c [id=f689bfaf5bea47b9a008c2657e3e15eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.089990 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 1f4d6be016284dc796945270178b9ee1 (DEFAULT_TABLE table=default.Table_9bc32b4aab234a4e8bab3910a4a3102b [id=cb4133ffe2c44743bdafdfafb7899843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.090484 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a079e985358040e78455ac8cb6e49f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.090797 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1b4da58289342d991cdf278005b96a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.091962 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4d6be016284dc796945270178b9ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.081194 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 50ac647b1d704142b94da8d506319466 (DEFAULT_TABLE table=default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.097919 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 93003ef3621a4f33acbc7eff54a796e3 (DEFAULT_TABLE table=default.Table_a3e83f37adb44078818e834e91f4e340 [id=a409c5484b27429f8e269705f3aac00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.101763 1165 ts_tablet_manager.cc:1918] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.102118 1165 log.cc:1199] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/93f449c80b604a6498f6fd1d1ba4770a
I20250625 13:52:01.104002 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ac647b1d704142b94da8d506319466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.104501 1165 ts_tablet_manager.cc:1939] T 93f449c80b604a6498f6fd1d1ba4770a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:01.109396 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 426) took 1857 ms. Trace:
I20250625 13:52:01.109580 1165 rpcz_store.cc:276] 0625 13:51:59.251680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.251735 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:00.899381 (+1647646us) ts_tablet_manager.cc:1145] Deleting tablet 93f449c80b604a6498f6fd1d1ba4770a
0625 13:52:00.899397 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.056070 (+156673us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.101626 (+ 45556us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.109333 (+ 7707us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647356}
I20250625 13:52:01.110175 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 427) took 1853 ms. Trace:
I20250625 13:52:01.110313 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 93f449c80b604a6498f6fd1d1ba4770a (table default.Table_7ccc261a6367449583a5c30495bfe2a9 [id=5436e1b638d14b00a66842a43cc5c085]) successfully deleted
I20250625 13:52:01.110327 1165 rpcz_store.cc:276] 0625 13:51:59.256711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.256763 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:01.109994 (+1853231us) ts_tablet_manager.cc:1145] Deleting tablet d26820eaba98401da3a8b5348be21fc6
0625 13:52:01.110004 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.110112 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853007}
I20250625 13:52:01.110705 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 430) took 1820 ms. Trace:
I20250625 13:52:01.110801 1165 rpcz_store.cc:276] 0625 13:51:59.290261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.297270 (+ 7009us) service_pool.cc:225] Handling call
0625 13:52:01.110546 (+1813276us) ts_tablet_manager.cc:1145] Deleting tablet fd39216b01054ff38ec9175d7a2f87f2
0625 13:52:01.110554 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.110653 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809806}
W20250625 13:52:01.110888 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d26820eaba98401da3a8b5348be21fc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d26820eaba98401da3a8b5348be21fc6 already in progress: creating tablet
I20250625 13:52:01.110900 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93003ef3621a4f33acbc7eff54a796e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.111161 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 434) took 1765 ms. Trace:
I20250625 13:52:01.111256 1165 rpcz_store.cc:276] 0625 13:51:59.345568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.345623 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:01.111033 (+1765410us) ts_tablet_manager.cc:1145] Deleting tablet ecb9a98e402b4da8a9c2bd8317cef532
0625 13:52:01.111043 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.111117 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1765117}
I20250625 13:52:01.111646 1165 tablet_replica.cc:331] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.111876 1165 raft_consensus.cc:2241] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.112109 1165 raft_consensus.cc:2270] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:01.112303 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet fd39216b01054ff38ec9175d7a2f87f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd39216b01054ff38ec9175d7a2f87f2 already in progress: creating tablet
W20250625 13:52:01.112545 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ecb9a98e402b4da8a9c2bd8317cef532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb9a98e402b4da8a9c2bd8317cef532 already in progress: creating tablet
I20250625 13:52:01.112721 1165 ts_tablet_manager.cc:1905] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.163235 1165 ts_tablet_manager.cc:1918] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.166146 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d139478986e346e2ac833ce353cff7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.166258 1046 raft_consensus.cc:3058] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.167430 1165 log.cc:1199] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/f73f420a4d7a438eadde3d6af64f0a11
I20250625 13:52:01.169641 1165 ts_tablet_manager.cc:1939] T f73f420a4d7a438eadde3d6af64f0a11 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:01.174126 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 439) took 1798 ms. Trace:
I20250625 13:52:01.177767 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 944759869ced45f09dfad8959b32edff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.178032 1165 rpcz_store.cc:276] 0625 13:51:59.375264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.378464 (+ 3200us) service_pool.cc:225] Handling call
0625 13:52:01.111555 (+1733091us) ts_tablet_manager.cc:1145] Deleting tablet f73f420a4d7a438eadde3d6af64f0a11
0625 13:52:01.111562 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.127934 (+ 16372us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.163054 (+ 35120us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.174084 (+ 11030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732772}
I20250625 13:52:01.179752 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 442) took 1787 ms. Trace:
I20250625 13:52:01.179795 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet f73f420a4d7a438eadde3d6af64f0a11 (table default.Table_9e86330e77034242a9aa4fbd274c3151 [id=0b06b619484e459f949d8acb2bb19512]) successfully deleted
I20250625 13:52:01.182332 1171 tablet_bootstrap.cc:492] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.182579 1171 ts_tablet_manager.cc:1397] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.456s user 0.000s sys 0.009s
I20250625 13:52:01.182521 1165 rpcz_store.cc:276] 0625 13:51:59.392354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.392546 (+ 192us) service_pool.cc:225] Handling call
0625 13:52:01.179583 (+1787037us) ts_tablet_manager.cc:1145] Deleting tablet 6f5780d88dea41e7b855b41f12e96c52
0625 13:52:01.179597 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.179715 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786774}
W20250625 13:52:01.187011 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6f5780d88dea41e7b855b41f12e96c52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f5780d88dea41e7b855b41f12e96c52 already in progress: creating tablet
I20250625 13:52:01.188094 881 tablet_service.cc:1468] Processing CreateTablet for tablet 39da3ebbe55f4c4fab4f8af444657b9b (DEFAULT_TABLE table=default.Table_fed63050407b40569f0a8911e9d1fd21 [id=6e495c98f9344b37afc59f947f596b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.188611 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58a532e889244b088562081e444234a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.188665 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 438) took 1813 ms. Trace:
I20250625 13:52:01.188778 1165 rpcz_store.cc:276] 0625 13:51:59.375141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.403439 (+ 28298us) service_pool.cc:225] Handling call
0625 13:52:01.188496 (+1785057us) ts_tablet_manager.cc:1145] Deleting tablet 430c3bed360b4f32855a8adb99076608
0625 13:52:01.188509 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.188631 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1684183}
W20250625 13:52:01.189220 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 430c3bed360b4f32855a8adb99076608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 430c3bed360b4f32855a8adb99076608 already in progress: creating tablet
I20250625 13:52:01.191289 1165 tablet_replica.cc:331] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.192004 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84b93a9079e45bab55353734ebe7cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.192279 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39da3ebbe55f4c4fab4f8af444657b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.192513 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f4d6be016284dc796945270178b9ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.193882 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c737daaede054d09a0c07564157423a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.194407 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99fe5ad690049029b8325dd38a11580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.194554 895 tablet_service.cc:1468] Processing CreateTablet for tablet e1b4da58289342d991cdf278005b96a0 (DEFAULT_TABLE table=default.Table_ff33e931cb1c4e44a12f7dcb8ddad00c [id=f689bfaf5bea47b9a008c2657e3e15eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.195060 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1b4da58289342d991cdf278005b96a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.195289 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d139478986e346e2ac833ce353cff7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.195811 1165 raft_consensus.cc:2241] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.196017 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ac647b1d704142b94da8d506319466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.196024 1165 raft_consensus.cc:2270] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.196606 1165 ts_tablet_manager.cc:1905] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.196714 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc350f8573f74dd88dcbc4c9a7c192c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.197569 879 tablet_service.cc:1468] Processing CreateTablet for tablet 93003ef3621a4f33acbc7eff54a796e3 (DEFAULT_TABLE table=default.Table_a3e83f37adb44078818e834e91f4e340 [id=a409c5484b27429f8e269705f3aac00e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:01.197703 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254afffd283449fdaf5e190ba3028f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.198120 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93003ef3621a4f33acbc7eff54a796e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.198787 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a079e985358040e78455ac8cb6e49f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.199450 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 944759869ced45f09dfad8959b32edff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.199633 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8d2ab8f91a045a3a39f3d09459f2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.199882 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54cc2d0cf95b41d6b36483af5436fcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.200439 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04479dfb34584f0891c604ef72bf93b5" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:01.201476 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d524089e6ab044ec961cfc9f587e8f86" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:01.201478 711 leader_election.cc:343] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.201678 711 leader_election.cc:304] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.202060 1669 raft_consensus.cc:2747] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.202492 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0a59da2129141a698692c4b31609d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.211959 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4b428ae7bc49fcbc37e9d7d238ec21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:01.216882 1165 ts_tablet_manager.cc:1918] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.231523 1165 log.cc:1199] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/cf1664afbab647b3baeed6391196bed9
I20250625 13:52:01.232210 1165 ts_tablet_manager.cc:1939] T cf1664afbab647b3baeed6391196bed9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:01.233417 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 444) took 1651 ms. Trace:
I20250625 13:52:01.233673 1165 rpcz_store.cc:276] 0625 13:51:59.582380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.582436 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:01.191218 (+1608782us) ts_tablet_manager.cc:1145] Deleting tablet cf1664afbab647b3baeed6391196bed9
0625 13:52:01.191228 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.212896 (+ 21668us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.216770 (+ 3874us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.233339 (+ 16569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608557}
I20250625 13:52:01.234369 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 445) took 1646 ms. Trace:
I20250625 13:52:01.234396 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet cf1664afbab647b3baeed6391196bed9 (table default.Table_98ae9c7310a24e8d988b919a5ec496e7 [id=05b9dac188b34d0b8022defddccbff26]) successfully deleted
I20250625 13:52:01.234531 1165 rpcz_store.cc:276] 0625 13:51:59.588103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.588154 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:01.234139 (+1645985us) ts_tablet_manager.cc:1145] Deleting tablet 413afd7b62d24fe8b9eceadd57e29fc8
0625 13:52:01.234150 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.234320 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1645770}
I20250625 13:52:01.234915 1165 tablet_replica.cc:331] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.235150 1165 raft_consensus.cc:2241] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.235361 1165 raft_consensus.cc:2270] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:01.235452 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 413afd7b62d24fe8b9eceadd57e29fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 413afd7b62d24fe8b9eceadd57e29fc8 already in progress: creating tablet
I20250625 13:52:01.236069 1165 ts_tablet_manager.cc:1905] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.237936 1518 raft_consensus.cc:491] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.239219 983 leader_election.cc:304] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.239154 1518 raft_consensus.cc:513] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:01.241568 849 consensus_peers.cc:487] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 600d711975a74f66b9144758ad0dcfb0. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:01.251704 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7641869a0bb463698244ff445d87a1f" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:01.252300 912 raft_consensus.cc:2466] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:01.256021 1165 ts_tablet_manager.cc:1918] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.256186 1165 log.cc:1199] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/c911d9ecbded417daf70e85c7cbcd52e
I20250625 13:52:01.267843 1165 ts_tablet_manager.cc:1939] T c911d9ecbded417daf70e85c7cbcd52e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:01.268777 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 446) took 1676 ms. Trace:
I20250625 13:52:01.268980 1165 rpcz_store.cc:276] 0625 13:51:59.591839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.591892 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:01.234827 (+1642935us) ts_tablet_manager.cc:1145] Deleting tablet c911d9ecbded417daf70e85c7cbcd52e
0625 13:52:01.234836 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.250074 (+ 15238us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.255883 (+ 5809us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.268734 (+ 12851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642744}
I20250625 13:52:01.269616 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet c911d9ecbded417daf70e85c7cbcd52e (table default.Table_a90cc30efd7145fdbaa7ebdc5db04107 [id=dec0cb5442804b6398a22dbd8c9b0f3a]) successfully deleted
I20250625 13:52:01.270143 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 447) took 1655 ms. Trace:
I20250625 13:52:01.270308 1165 rpcz_store.cc:276] 0625 13:51:59.614473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.614534 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:01.269975 (+1655441us) ts_tablet_manager.cc:1145] Deleting tablet abd6f9121a9d4b39adde1b880a875e23
0625 13:52:01.269986 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.270109 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655196}
W20250625 13:52:01.270835 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet abd6f9121a9d4b39adde1b880a875e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abd6f9121a9d4b39adde1b880a875e23 already in progress: creating tablet
I20250625 13:52:01.271225 1165 tablet_replica.cc:331] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.271517 1165 raft_consensus.cc:2241] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.271749 1165 raft_consensus.cc:2270] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.272531 1165 ts_tablet_manager.cc:1905] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.284109 1168 ts_tablet_manager.cc:1918] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:01.284440 1168 log.cc:1199] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/85c72f9ae92a4537979fb479eb4467c4
I20250625 13:52:01.298571 1165 ts_tablet_manager.cc:1918] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:01.301121 985 leader_election.cc:343] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b7641869a0bb463698244ff445d87a1f
I20250625 13:52:01.312888 1165 log.cc:1199] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/41a7862601f04fd7a938f7e5d5a4c0f9
I20250625 13:52:01.313390 1165 ts_tablet_manager.cc:1939] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:01.314289 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 448) took 1694 ms. Trace:
I20250625 13:52:01.315011 1165 rpcz_store.cc:276] 0625 13:51:59.619753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.619805 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:01.271154 (+1651349us) ts_tablet_manager.cc:1145] Deleting tablet 41a7862601f04fd7a938f7e5d5a4c0f9
0625 13:52:01.271163 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.276481 (+ 5318us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.298451 (+ 21970us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.314230 (+ 15779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651117}
I20250625 13:52:01.319859 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 41a7862601f04fd7a938f7e5d5a4c0f9 (table default.Table_478b0037c80e40048d4f8ebc92748f34 [id=13d38d05aad34e22884b9e07610b0e87]) successfully deleted
I20250625 13:52:01.320451 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 449) took 1662 ms. Trace:
I20250625 13:52:01.323002 1165 rpcz_store.cc:276] 0625 13:51:59.657843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.657899 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:01.320242 (+1662343us) ts_tablet_manager.cc:1145] Deleting tablet 77c50e1ebb9042ef8f0a651ba7186b74
0625 13:52:01.320260 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.320394 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1662107}
W20250625 13:52:01.323758 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 77c50e1ebb9042ef8f0a651ba7186b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77c50e1ebb9042ef8f0a651ba7186b74 already in progress: creating tablet
I20250625 13:52:01.324306 1165 tablet_replica.cc:331] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.324254 1518 leader_election.cc:290] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.324602 1165 raft_consensus.cc:2241] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.324898 1165 raft_consensus.cc:2270] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.325577 1165 ts_tablet_manager.cc:1905] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.325816 983 leader_election.cc:304] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:01.334072 1166 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.024s sys 0.076s
W20250625 13:52:01.337605 1166 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.560s user 0.024s sys 0.076s
I20250625 13:52:01.338553 1166 tablet_bootstrap.cc:492] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.338790 1166 ts_tablet_manager.cc:1397] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.581s user 0.026s sys 0.076s
I20250625 13:52:01.339670 1166 raft_consensus.cc:357] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.339949 1166 raft_consensus.cc:383] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.340051 1166 raft_consensus.cc:738] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.340335 1166 consensus_queue.cc:260] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:01.341351 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 89123.3us
I20250625 13:52:01.347698 1166 ts_tablet_manager.cc:1428] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
W20250625 13:52:01.350651 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 84153.8us
I20250625 13:52:01.353933 1518 raft_consensus.cc:2802] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:01.355003 1518 raft_consensus.cc:491] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:01.355129 1518 raft_consensus.cc:3058] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.356593 1166 tablet_bootstrap.cc:492] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.358397 1166 tablet_bootstrap.cc:654] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.360874 1165 ts_tablet_manager.cc:1918] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.361140 1165 log.cc:1199] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/cd4f0857f52449e1b26fde09c6c7c9d4
I20250625 13:52:01.362747 1165 ts_tablet_manager.cc:1939] T cd4f0857f52449e1b26fde09c6c7c9d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:01.363634 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 450) took 1658 ms. Trace:
I20250625 13:52:01.363811 1165 rpcz_store.cc:276] 0625 13:51:59.705382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.705443 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:01.324211 (+1618768us) ts_tablet_manager.cc:1145] Deleting tablet cd4f0857f52449e1b26fde09c6c7c9d4
0625 13:52:01.324223 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.329446 (+ 5223us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.360715 (+ 31269us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.363595 (+ 2880us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1611114}
I20250625 13:52:01.364391 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet cd4f0857f52449e1b26fde09c6c7c9d4 (table default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]) successfully deleted
I20250625 13:52:01.364802 1165 tablet_replica.cc:331] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.364897 1669 raft_consensus.cc:491] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.365084 1669 raft_consensus.cc:513] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:01.365155 1165 raft_consensus.cc:2241] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.365617 1165 raft_consensus.cc:2270] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.365846 1669 leader_election.cc:290] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:01.367234 1165 ts_tablet_manager.cc:1905] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.370540 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb1c7a85da844d34be805843bc60d350" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:01.372946 711 leader_election.cc:343] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.386966 1518 raft_consensus.cc:513] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.389139 1166 tablet_bootstrap.cc:492] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.389333 1166 ts_tablet_manager.cc:1397] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250625 13:52:01.390412 1166 raft_consensus.cc:357] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.390657 1166 raft_consensus.cc:383] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.390769 1166 raft_consensus.cc:738] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.391088 1166 consensus_queue.cc:260] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.391973 1166 ts_tablet_manager.cc:1428] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250625 13:52:01.392509 1166 tablet_bootstrap.cc:492] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.394922 1166 tablet_bootstrap.cc:654] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.401093 1165 ts_tablet_manager.cc:1918] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.401643 1165 log.cc:1199] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/7d361e73fe3d4ea2bb6735f814345ebb
I20250625 13:52:01.402135 1165 ts_tablet_manager.cc:1939] T 7d361e73fe3d4ea2bb6735f814345ebb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:52:01.402312 985 leader_election.cc:343] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b7641869a0bb463698244ff445d87a1f
I20250625 13:52:01.406114 1166 tablet_bootstrap.cc:492] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.406351 1166 ts_tablet_manager.cc:1397] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20250625 13:52:01.407217 1166 raft_consensus.cc:357] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.285049 1168 ts_tablet_manager.cc:1939] T 85c72f9ae92a4537979fb479eb4467c4 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:01.371532 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb1c7a85da844d34be805843bc60d350" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:01.407761 1166 raft_consensus.cc:383] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.407840 1166 raft_consensus.cc:738] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.407970 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7641869a0bb463698244ff445d87a1f" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:01.408200 913 raft_consensus.cc:3058] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.408192 1166 consensus_queue.cc:260] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.408563 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 203) took 6023 ms. Trace:
I20250625 13:52:01.408671 1166 ts_tablet_manager.cc:1428] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:01.408692 1168 rpcz_store.cc:276] 0625 13:51:55.385313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.385528 (+ 215us) service_pool.cc:225] Handling call
0625 13:52:00.820046 (+5434518us) ts_tablet_manager.cc:1145] Deleting tablet 85c72f9ae92a4537979fb479eb4467c4
0625 13:52:00.820055 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:00.906214 (+ 86159us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.283887 (+377673us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.408519 (+124632us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5428497}
I20250625 13:52:01.409000 1166 tablet_bootstrap.cc:492] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.409288 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 204) took 5958 ms. Trace:
I20250625 13:52:01.409405 1168 rpcz_store.cc:276] 0625 13:51:55.450501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.450757 (+ 256us) service_pool.cc:225] Handling call
0625 13:52:01.409138 (+5958381us) ts_tablet_manager.cc:1145] Deleting tablet f00e72e103f443e886f0df690cff4b6f
0625 13:52:01.409147 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.409246 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5955365}
I20250625 13:52:01.409845 1168 tablet_replica.cc:331] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:01.410396 1168 raft_consensus.cc:2241] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.410579 1168 raft_consensus.cc:2270] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.411116 1168 ts_tablet_manager.cc:1905] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.411216 1166 tablet_bootstrap.cc:654] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:01.411885 711 leader_election.cc:343] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.412137 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 454) took 1629 ms. Trace:
I20250625 13:52:01.412092 711 leader_election.cc:304] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.412467 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 85c72f9ae92a4537979fb479eb4467c4 (table default.Table_8fc2c990732d4e658580c2bc2cf67f4d [id=75b34bf006e040d18266d0c90986afeb]) successfully deleted
I20250625 13:52:01.413026 1669 raft_consensus.cc:2747] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.400267 1171 raft_consensus.cc:357] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.415570 1171 raft_consensus.cc:383] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.415660 1171 raft_consensus.cc:738] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.415863 1171 consensus_queue.cc:260] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.416365 1171 ts_tablet_manager.cc:1428] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.233s user 0.001s sys 0.000s
I20250625 13:52:01.417623 913 raft_consensus.cc:2466] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:01.418450 1171 tablet_bootstrap.cc:492] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:01.418673 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu pid 1222
I20250625 13:52:01.419358 1165 rpcz_store.cc:276] 0625 13:51:59.782935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.782997 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:01.364731 (+1581734us) ts_tablet_manager.cc:1145] Deleting tablet 7d361e73fe3d4ea2bb6735f814345ebb
0625 13:52:01.364740 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.373079 (+ 8339us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.400949 (+ 27870us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.412071 (+ 11122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1581515}
I20250625 13:52:01.420189 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 7d361e73fe3d4ea2bb6735f814345ebb (table default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]) successfully deleted
I20250625 13:52:01.420979 1171 tablet_bootstrap.cc:654] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.422389 1687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52994 (request call id 454) took 1182 ms. Trace:
I20250625 13:52:01.422516 1687 rpcz_store.cc:276] 0625 13:52:00.240213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.240266 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:01.422348 (+1182082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:52:00.328377 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0625 13:52:00.328512 (+ 135us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0625 13:52:00.328520 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
0625 13:52:00.328526 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0625 13:52:00.328587 (+ 61us) alter_schema_op.cc:128] START. Timestamp: 1750859520328542
0625 13:52:00.328589 (+ 2us) op_driver.cc:348] REPLICATION: starting
0625 13:52:00.328751 (+ 162us) log.cc:844] Serialized 173 byte log entry
0625 13:52:00.738808 (+410057us) op_driver.cc:464] REPLICATION: finished
0625 13:52:00.827123 (+ 88315us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0625 13:52:00.839498 (+ 12375us) log.cc:844] Serialized 5 byte log entry
0625 13:52:01.402069 (+562571us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.422240 (+ 20171us) log.cc:844] Serialized 14 byte log entry
0625 13:52:01.422293 (+ 53us) alter_schema_op.cc:73] Released schema lock
0625 13:52:01.422301 (+ 8us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88242,"prepare.queue_time_us":545,"prepare.run_cpu_time_us":482,"prepare.run_wall_time_us":482,"raft.queue_time_us":401,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":353,"replication_time_us":410181,"thread_start_us":84486,"threads_started":1}]]}
I20250625 13:52:01.427081 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 456) took 1615 ms. Trace:
I20250625 13:52:01.429222 1165 rpcz_store.cc:276] 0625 13:51:59.812092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.818275 (+ 6183us) service_pool.cc:225] Handling call
0625 13:52:01.426856 (+1608581us) ts_tablet_manager.cc:1145] Deleting tablet bb648c45ea344fb3bac765c9e06db2b1
0625 13:52:01.426872 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.427011 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1585039}
W20250625 13:52:01.429983 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bb648c45ea344fb3bac765c9e06db2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb648c45ea344fb3bac765c9e06db2b1 already in progress: creating tablet
I20250625 13:52:01.430509 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 455) took 1621 ms. Trace:
I20250625 13:52:01.430994 1165 rpcz_store.cc:276] 0625 13:51:59.809304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.810326 (+ 1022us) service_pool.cc:225] Handling call
0625 13:52:01.430353 (+1620027us) ts_tablet_manager.cc:1145] Deleting tablet 4c9efc7631df4ad29112a41f229ae2e9
0625 13:52:01.430363 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.430460 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1588425}
W20250625 13:52:01.431578 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4c9efc7631df4ad29112a41f229ae2e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c9efc7631df4ad29112a41f229ae2e9 already in progress: creating tablet
I20250625 13:52:01.433947 983 leader_election.cc:304] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:01.437803 1518 leader_election.cc:290] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.451473 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 457) took 1604 ms. Trace:
I20250625 13:52:01.452508 1165 rpcz_store.cc:276] 0625 13:51:59.846877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.847381 (+ 504us) service_pool.cc:225] Handling call
0625 13:52:01.434285 (+1586904us) ts_tablet_manager.cc:1145] Deleting tablet dbd2120b4b084691b6604305773e74f9
0625 13:52:01.451222 (+ 16937us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.451401 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586470}
I20250625 13:52:01.453276 1518 raft_consensus.cc:2802] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:01.453675 1518 raft_consensus.cc:695] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:01.454113 1518 consensus_queue.cc:237] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:01.454309 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet dbd2120b4b084691b6604305773e74f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbd2120b4b084691b6604305773e74f9 already in progress: creating tablet
I20250625 13:52:01.455370 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 462) took 1563 ms. Trace:
I20250625 13:52:01.457167 1165 rpcz_store.cc:276] 0625 13:51:59.891734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.891787 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:01.454665 (+1562878us) ts_tablet_manager.cc:1145] Deleting tablet dcf7fab53cae49908d2952cdaa07943a
0625 13:52:01.454674 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.454775 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562676}
I20250625 13:52:01.466689 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_ebb1cd6aa381410a82cac38e7197353c [id=d7cbe338b2554a6e99b07325632aaa33] alter complete (version 3)
I20250625 13:52:01.459220 1166 tablet_bootstrap.cc:492] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.467770 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.468001 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 04479dfb34584f0891c604ef72bf93b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.471266 1166 ts_tablet_manager.cc:1397] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
I20250625 13:52:01.472785 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet b7641869a0bb463698244ff445d87a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:01.473235 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 3016763b9e294b9da2861fe97e57e241 (table default.Table_dfa5acc20d2d4b34ba0a3503952923f5 [id=0f2c02514ee0468b9aaa8095831374ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3016763b9e294b9da2861fe97e57e241 consensus state has no leader
I20250625 13:52:01.472877 1166 raft_consensus.cc:357] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.473531 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]). Expected version 1 got 0
I20250625 13:52:01.473927 1166 raft_consensus.cc:383] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.474108 1166 raft_consensus.cc:738] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.474395 1166 consensus_queue.cc:260] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.474965 1166 ts_tablet_manager.cc:1428] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:01.475322 1166 tablet_bootstrap.cc:492] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.475548 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet d524089e6ab044ec961cfc9f587e8f86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.477226 743 tablet_service.cc:1515] Processing DeleteTablet for tablet f033bfa4925b4d1c971789eb77efd92c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.478754 1166 tablet_bootstrap.cc:654] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:01.479640 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet dcf7fab53cae49908d2952cdaa07943a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcf7fab53cae49908d2952cdaa07943a already in progress: creating tablet
I20250625 13:52:01.480202 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 473) took 1547 ms. Trace:
I20250625 13:52:01.482388 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.483608 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a29eb5d240db4787866630bc39e35db5" } new_table_name: "default.Table_1fafa11a1445445081345630d89f80dc" modify_external_catalogs: true
W20250625 13:52:01.489257 851 consensus_peers.cc:487] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41a7862601f04fd7a938f7e5d5a4c0f9. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:01.490767 1165 rpcz_store.cc:276] 0625 13:51:59.932874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.932924 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:01.479993 (+1547069us) ts_tablet_manager.cc:1145] Deleting tablet 077e8c001cc646559b5268d4a48c0ad6
0625 13:52:01.480007 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.480137 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546848}
W20250625 13:52:01.491458 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 077e8c001cc646559b5268d4a48c0ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 077e8c001cc646559b5268d4a48c0ad6 already in progress: creating tablet
I20250625 13:52:01.492240 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 482) took 1396 ms. Trace:
I20250625 13:52:01.492417 1165 rpcz_store.cc:276] 0625 13:52:00.096166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.096486 (+ 320us) service_pool.cc:225] Handling call
0625 13:52:01.491838 (+1395352us) ts_tablet_manager.cc:1145] Deleting tablet 7efc8df3c8e24020b073a1b61bc9e0a0
0625 13:52:01.491861 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.492183 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1332592}
W20250625 13:52:01.492908 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7efc8df3c8e24020b073a1b61bc9e0a0 already in progress: creating tablet
I20250625 13:52:01.493402 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 483) took 1397 ms. Trace:
I20250625 13:52:01.493544 1165 rpcz_store.cc:276] 0625 13:52:00.096280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.096721 (+ 441us) service_pool.cc:225] Handling call
0625 13:52:01.493232 (+1396511us) ts_tablet_manager.cc:1145] Deleting tablet 829221b8a79f460d90dd24be6a2d4c43
0625 13:52:01.493244 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.493343 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333551}
W20250625 13:52:01.494058 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 829221b8a79f460d90dd24be6a2d4c43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829221b8a79f460d90dd24be6a2d4c43 already in progress: creating tablet
W20250625 13:52:01.499352 1170 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.174s sys 0.316s
W20250625 13:52:01.499476 1170 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.612s user 0.174s sys 0.316s
I20250625 13:52:01.500428 1170 tablet_bootstrap.cc:492] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.500630 1170 ts_tablet_manager.cc:1397] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.617s user 0.176s sys 0.316s
I20250625 13:52:01.501786 1170 raft_consensus.cc:357] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:01.501964 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 83721us
I20250625 13:52:01.502017 1170 raft_consensus.cc:383] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.502103 1170 raft_consensus.cc:738] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.502310 1170 consensus_queue.cc:260] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.502789 1170 ts_tablet_manager.cc:1428] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:52:01.502914 1704 raft_consensus.cc:491] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.503100 1170 tablet_bootstrap.cc:492] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.503120 1704 raft_consensus.cc:513] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.506156 1704 leader_election.cc:290] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:01.506356 985 leader_election.cc:343] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 02f2a2b595bc43309a29b3f1bc5d3766
I20250625 13:52:01.504875 1170 tablet_bootstrap.cc:654] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.507179 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02f2a2b595bc43309a29b3f1bc5d3766" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:01.508261 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc350f8573f74dd88dcbc4c9a7c192c4 consensus state has no leader
I20250625 13:52:01.508790 983 leader_election.cc:304] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.509779 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_93d4946c778542148fe4644b897607be" } modify_external_catalogs: true
I20250625 13:52:01.509994 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_93d4946c778542148fe4644b897607be" } modify_external_catalogs: true
I20250625 13:52:01.511432 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_20d0eb2c546641379a1f7ff2c77427d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.511999 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20d0eb2c546641379a1f7ff2c77427d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.513831 1704 raft_consensus.cc:2747] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.515169 1710 tablet_replica.cc:331] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:01.516245 1168 ts_tablet_manager.cc:1918] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.516408 1168 log.cc:1199] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/09a8e15d57f14702ba0b5b52a94fc7c4
I20250625 13:52:01.516827 1168 ts_tablet_manager.cc:1939] T 09a8e15d57f14702ba0b5b52a94fc7c4 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:01.516889 1171 tablet_bootstrap.cc:492] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.517676 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 205) took 6041 ms. Trace:
I20250625 13:52:01.517777 1168 rpcz_store.cc:276] 0625 13:51:55.475782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.476197 (+ 415us) service_pool.cc:225] Handling call
0625 13:52:01.409775 (+5933578us) ts_tablet_manager.cc:1145] Deleting tablet 09a8e15d57f14702ba0b5b52a94fc7c4
0625 13:52:01.409785 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.414292 (+ 4507us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.516102 (+101810us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.517614 (+ 1512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5933228}
I20250625 13:52:01.518215 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 09a8e15d57f14702ba0b5b52a94fc7c4 (table default.Table_abfcbd249e4a4189aba6f16b5a66a0e7 [id=7d3d8e359ea945cc9495eb2454e1d85b]) successfully deleted
I20250625 13:52:01.524995 1170 tablet_bootstrap.cc:492] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.525185 1170 ts_tablet_manager.cc:1397] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250625 13:52:01.526311 1170 raft_consensus.cc:357] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.526546 1170 raft_consensus.cc:383] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.526639 1170 raft_consensus.cc:738] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.526782 1171 ts_tablet_manager.cc:1397] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.108s user 0.006s sys 0.003s
I20250625 13:52:01.526925 1170 consensus_queue.cc:260] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.527354 1168 tablet_replica.cc:331] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:01.527426 1170 ts_tablet_manager.cc:1428] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250625 13:52:01.527755 1170 tablet_bootstrap.cc:492] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.527791 1171 raft_consensus.cc:357] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.528241 1168 raft_consensus.cc:2241] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:01.529719 1170 tablet_bootstrap.cc:654] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.530140 1168 pending_rounds.cc:70] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:01.530273 1168 pending_rounds.cc:77] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520577564020736 op_type: NO_OP noop_request { }
I20250625 13:52:01.530582 1168 raft_consensus.cc:2887] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:01.530735 1168 raft_consensus.cc:2270] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.531432 1168 ts_tablet_manager.cc:1905] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.532771 1528 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 13:52:01.533790 1170 tablet_bootstrap.cc:492] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.533957 1170 ts_tablet_manager.cc:1397] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250625 13:52:01.534523 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 484) took 1438 ms. Trace:
I20250625 13:52:01.534657 1165 rpcz_store.cc:276] 0625 13:52:00.096376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.096778 (+ 402us) service_pool.cc:225] Handling call
0625 13:52:01.494317 (+1397539us) ts_tablet_manager.cc:1145] Deleting tablet 150af49427ae47b98e9b2077bbbe1e2b
0625 13:52:01.534288 (+ 39971us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x61800000d518. stack: 00007f4161830336 00007f41618300f0 00007f41608263ad 0000000000971004 0000000000970fcd 000000000097099a 00007f4161818c3d 00007f4161819cae 00007f4174eae80d 00007f4174e300f3 00007f417107d4f6 00007f417107e879 00007f41710b11c4 00007f4171092c20 00007f41710929ad 0000000000a7573c
0625 13:52:01.534302 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.534454 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91129728,"tablet-delete.queue_time_us":1330792}
I20250625 13:52:01.534874 1170 raft_consensus.cc:357] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.535049 1170 raft_consensus.cc:383] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.535151 1170 raft_consensus.cc:738] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:01.535881 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 150af49427ae47b98e9b2077bbbe1e2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 150af49427ae47b98e9b2077bbbe1e2b already in progress: creating tablet
I20250625 13:52:01.535434 1170 consensus_queue.cc:260] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.528291 1171 raft_consensus.cc:383] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.536262 1171 raft_consensus.cc:738] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.536398 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 486) took 1352 ms. Trace:
I20250625 13:52:01.536463 1170 ts_tablet_manager.cc:1428] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:01.536502 1165 rpcz_store.cc:276] 0625 13:52:00.183602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.183656 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:01.536221 (+1352565us) ts_tablet_manager.cc:1145] Deleting tablet cbf7e30cdff245208dacc6963fe88c7d
0625 13:52:01.536233 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.536337 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1352312}
I20250625 13:52:01.536846 1170 tablet_bootstrap.cc:492] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:01.536983 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet cbf7e30cdff245208dacc6963fe88c7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbf7e30cdff245208dacc6963fe88c7d already in progress: creating tablet
I20250625 13:52:01.537417 1165 tablet_replica.cc:331] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.536664 1171 consensus_queue.cc:260] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.537629 1165 raft_consensus.cc:2241] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.537812 1165 raft_consensus.cc:2270] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.538406 1165 ts_tablet_manager.cc:1905] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.538647 1170 tablet_bootstrap.cc:654] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.543017 1171 ts_tablet_manager.cc:1428] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 13:52:01.544430 1710 raft_consensus.cc:2241] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:01.544847 1710 pending_rounds.cc:70] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:01.544930 1710 pending_rounds.cc:77] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520599879987200 op_type: NO_OP noop_request { }
I20250625 13:52:01.545079 1710 raft_consensus.cc:2887] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:01.545183 1710 raft_consensus.cc:2270] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.545786 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4dd664d07c7f4b22a44352da05e41e94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:01.546483 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 991bb428def545fd979e9f1a80ce4d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:01.546509 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dd664d07c7f4b22a44352da05e41e94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.546553 1171 tablet_bootstrap.cc:492] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:01.549119 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_194c4cbe54834768a869225974bea8ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.549618 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_194c4cbe54834768a869225974bea8ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.551270 1166 tablet_bootstrap.cc:492] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.551460 1166 ts_tablet_manager.cc:1397] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.000s
I20250625 13:52:01.552438 1166 raft_consensus.cc:357] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.552664 1166 raft_consensus.cc:383] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.552747 1166 raft_consensus.cc:738] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.553006 1166 consensus_queue.cc:260] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.553570 1166 ts_tablet_manager.cc:1428] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:01.553936 1166 tablet_bootstrap.cc:492] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.556125 1710 ts_tablet_manager.cc:1905] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.563680 1710 ts_tablet_manager.cc:1918] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:01.563978 1710 log.cc:1199] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/b7641869a0bb463698244ff445d87a1f
I20250625 13:52:01.564538 1710 ts_tablet_manager.cc:1939] T b7641869a0bb463698244ff445d87a1f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:01.565124 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 063a793e0d7747668d2728fdc2160b5e
I20250625 13:52:01.565168 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet b7641869a0bb463698244ff445d87a1f (table default.Table_6e5d1377cfc84651afd53a78735468d8 [id=bd476a97393841899b4cdb426c39e6e9]) successfully deleted
I20250625 13:52:01.565650 589 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:54762 (request call id 634) took 1300 ms. Trace:
I20250625 13:52:01.565873 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 063a793e0d7747668d2728fdc2160b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.566552 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 063a793e0d7747668d2728fdc2160b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.566958 1710 tablet_replica.cc:331] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:01.565797 589 rpcz_store.cc:276] 0625 13:52:00.265614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.268027 (+ 2413us) service_pool.cc:225] Handling call
0625 13:52:00.268463 (+ 436us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 13:52:00.268608 (+ 145us) catalog_manager.cc:2790] Modifying in-memory table state
0625 13:52:00.268660 (+ 52us) catalog_manager.cc:2799] Locking tablets
0625 13:52:00.900587 (+631927us) catalog_manager.cc:2813] Removing table and tablets from system table
0625 13:52:01.564903 (+664316us) catalog_manager.cc:2831] Removing table from by-name map
0625 13:52:01.564998 (+ 95us) catalog_manager.cc:2847] Committing in-memory state
0625 13:52:01.565042 (+ 44us) catalog_manager.cc:2851] Aborting table tasks
0625 13:52:01.565613 (+ 571us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:52:01.544042 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 13:52:01.544164 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0625 13:52:01.544169 (+ 5us) write_op.cc:435] Acquired schema lock
0625 13:52:01.544172 (+ 3us) tablet.cc:660] Decoding operations
0625 13:52:01.544297 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0625 13:52:01.544310 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0625 13:52:01.544317 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0625 13:52:01.544438 (+ 121us) tablet.cc:699] Row locks acquired
0625 13:52:01.544442 (+ 4us) write_op.cc:260] PREPARE: finished
0625 13:52:01.544481 (+ 39us) write_op.cc:270] Start()
0625 13:52:01.544517 (+ 36us) write_op.cc:276] Timestamp: P: 1750859521544428 usec, L: 0
0625 13:52:01.544521 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 13:52:01.544737 (+ 216us) log.cc:844] Serialized 756 byte log entry
0625 13:52:01.562905 (+ 18168us) op_driver.cc:464] REPLICATION: finished
0625 13:52:01.564424 (+ 1519us) write_op.cc:301] APPLY: starting
0625 13:52:01.564444 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0625 13:52:01.564487 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0625 13:52:01.564524 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 13:52:01.564611 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 13:52:01.564646 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 13:52:01.564653 (+ 7us) write_op.cc:312] APPLY: finished
0625 13:52:01.564733 (+ 80us) log.cc:844] Serialized 29 byte log entry
0625 13:52:01.564797 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:52:01.564815 (+ 18us) write_op.cc:454] Released schema lock
0625 13:52:01.564821 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1457,"num_ops":2,"prepare.queue_time_us":642657,"raft.queue_time_us":14848,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":311,"replication_time_us":18353}]]}
I20250625 13:52:01.567893 1171 tablet_bootstrap.cc:654] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.569586 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 063a793e0d7747668d2728fdc2160b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.573854 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_30da1f016b424e85bcbae236fafd8060" } modify_external_catalogs: true
I20250625 13:52:01.574146 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_30da1f016b424e85bcbae236fafd8060" } modify_external_catalogs: true
I20250625 13:52:01.570991 1168 ts_tablet_manager.cc:1918] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:01.575187 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 055f88e118454e32baa578cae9aceb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.575464 1168 log.cc:1199] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/cf5a6951aba84f5fa8f6d76a2b7fcf2d
I20250625 13:52:01.576004 1168 ts_tablet_manager.cc:1939] T cf5a6951aba84f5fa8f6d76a2b7fcf2d P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:01.580272 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 206) took 6104 ms. Trace:
I20250625 13:52:01.581009 1168 rpcz_store.cc:276] 0625 13:51:55.475931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.476598 (+ 667us) service_pool.cc:225] Handling call
0625 13:52:01.527153 (+6050555us) ts_tablet_manager.cc:1145] Deleting tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d
0625 13:52:01.527166 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.553896 (+ 26730us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.570864 (+ 16968us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.580213 (+ 9349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6050338}
I20250625 13:52:01.581892 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet cf5a6951aba84f5fa8f6d76a2b7fcf2d (table default.Table_aac41d1ca7024d0c8808c6d1cb2c7b0e [id=2a70f0ade87f4f34bcbe23295f23234b]) successfully deleted
I20250625 13:52:01.582067 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4d2d6f05348e43339e7167a9cddca2ba
I20250625 13:52:01.582593 595 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:54762 (request call id 659) took 1016 ms. Trace:
I20250625 13:52:01.582811 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2d6f05348e43339e7167a9cddca2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.582695 595 rpcz_store.cc:276] 0625 13:52:00.565686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.613683 (+ 47997us) service_pool.cc:225] Handling call
0625 13:52:00.614144 (+ 461us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 13:52:00.614261 (+ 117us) catalog_manager.cc:2790] Modifying in-memory table state
0625 13:52:00.614332 (+ 71us) catalog_manager.cc:2799] Locking tablets
0625 13:52:00.900969 (+286637us) catalog_manager.cc:2813] Removing table and tablets from system table
0625 13:52:01.581897 (+680928us) catalog_manager.cc:2831] Removing table from by-name map
0625 13:52:01.581977 (+ 80us) catalog_manager.cc:2847] Committing in-memory state
0625 13:52:01.582014 (+ 37us) catalog_manager.cc:2851] Aborting table tasks
0625 13:52:01.582536 (+ 522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:52:01.575911 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 13:52:01.576055 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0625 13:52:01.576062 (+ 7us) write_op.cc:435] Acquired schema lock
0625 13:52:01.576065 (+ 3us) tablet.cc:660] Decoding operations
0625 13:52:01.576212 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0625 13:52:01.576231 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0625 13:52:01.576238 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0625 13:52:01.576378 (+ 140us) tablet.cc:699] Row locks acquired
0625 13:52:01.576383 (+ 5us) write_op.cc:260] PREPARE: finished
0625 13:52:01.576425 (+ 42us) write_op.cc:270] Start()
0625 13:52:01.576466 (+ 41us) write_op.cc:276] Timestamp: P: 1750859521576366 usec, L: 0
0625 13:52:01.576471 (+ 5us) op_driver.cc:348] REPLICATION: starting
0625 13:52:01.576705 (+ 234us) log.cc:844] Serialized 725 byte log entry
0625 13:52:01.579884 (+ 3179us) op_driver.cc:464] REPLICATION: finished
0625 13:52:01.581439 (+ 1555us) write_op.cc:301] APPLY: starting
0625 13:52:01.581456 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0625 13:52:01.581497 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0625 13:52:01.581531 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 13:52:01.581596 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 13:52:01.581644 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 13:52:01.581652 (+ 8us) write_op.cc:312] APPLY: finished
0625 13:52:01.581723 (+ 71us) log.cc:844] Serialized 29 byte log entry
0625 13:52:01.581784 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:52:01.581805 (+ 21us) write_op.cc:454] Released schema lock
0625 13:52:01.581811 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1500,"num_ops":2,"prepare.queue_time_us":674301,"raft.queue_time_us":42,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":246,"replication_time_us":3385}]]}
I20250625 13:52:01.583424 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2d6f05348e43339e7167a9cddca2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.584141 1171 tablet_bootstrap.cc:492] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.584340 1171 ts_tablet_manager.cc:1397] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.001s
I20250625 13:52:01.584896 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2d6f05348e43339e7167a9cddca2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.585254 1171 raft_consensus.cc:357] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.585517 1171 raft_consensus.cc:383] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.585670 1171 raft_consensus.cc:738] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.585986 1171 consensus_queue.cc:260] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.586839 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7700dabe9c4446908a9535b1b946a3e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.587486 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7700dabe9c4446908a9535b1b946a3e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:01.589326 849 consensus_peers.cc:487] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8149aea0ff7646d3b94c9159f56e48ef. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:01.590132 1171 ts_tablet_manager.cc:1428] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:01.590516 1171 tablet_bootstrap.cc:492] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:01.591617 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 207) took 6115 ms. Trace:
I20250625 13:52:01.591800 1168 rpcz_store.cc:276] 0625 13:51:55.476133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.476841 (+ 708us) service_pool.cc:225] Handling call
0625 13:52:01.591451 (+6114610us) ts_tablet_manager.cc:1145] Deleting tablet c666ff2d6b9b4ed4a18e370826c143fc
0625 13:52:01.591463 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.591567 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6114439}
W20250625 13:52:01.592550 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c666ff2d6b9b4ed4a18e370826c143fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c666ff2d6b9b4ed4a18e370826c143fc already in progress: creating tablet
I20250625 13:52:01.597095 1695 raft_consensus.cc:2747] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.599277 1510 raft_consensus.cc:491] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.599514 1510 raft_consensus.cc:513] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.600171 1510 leader_election.cc:290] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:01.603384 1710 raft_consensus.cc:2241] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.603874 1710 raft_consensus.cc:2270] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:01.604763 851 leader_election.cc:343] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 87327f74beb845768bbfd5ae6d2bcf88
W20250625 13:52:01.605221 849 leader_election.cc:343] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 87327f74beb845768bbfd5ae6d2bcf88
I20250625 13:52:01.605352 849 leader_election.cc:304] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:01.605767 1442 raft_consensus.cc:2747] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:01.606884 1710 ts_tablet_manager.cc:1905] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.609256 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ae4a105e80c1460cbb21e1bdb9b356b1
I20250625 13:52:01.609549 1170 tablet_bootstrap.cc:492] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.609701 1170 ts_tablet_manager.cc:1397] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.073s user 0.021s sys 0.034s
I20250625 13:52:01.609819 588 catalog_manager.cc:6652] ReplaceTablet: tablet ae4a105e80c1460cbb21e1bdb9b356b1 deleted and replaced by tablet c431abffd05c4136b7510c60eee2a8b1
I20250625 13:52:01.610005 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 with delete_type TABLET_DATA_DELETED (replaced by tablet c431abffd05c4136b7510c60eee2a8b1) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.610507 1170 raft_consensus.cc:357] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.610584 886 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 with delete_type TABLET_DATA_DELETED (replaced by tablet c431abffd05c4136b7510c60eee2a8b1) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.610694 1170 raft_consensus.cc:383] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.610765 1170 raft_consensus.cc:738] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.611006 1170 consensus_queue.cc:260] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.611380 743 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 with delete_type TABLET_DATA_DELETED (replaced by tablet c431abffd05c4136b7510c60eee2a8b1) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.611524 1170 ts_tablet_manager.cc:1428] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250625 13:52:01.611829 1170 tablet_bootstrap.cc:492] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.612691 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d85cdadfd694486590092a7b6f61f44c
I20250625 13:52:01.613174 593 catalog_manager.cc:6652] ReplaceTablet: tablet d85cdadfd694486590092a7b6f61f44c deleted and replaced by tablet 9aadc67058554e87bb58d14b911333c4
I20250625 13:52:01.613498 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d85cdadfd694486590092a7b6f61f44c with delete_type TABLET_DATA_DELETED (replaced by tablet 9aadc67058554e87bb58d14b911333c4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.613755 1170 tablet_bootstrap.cc:654] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.613920 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet d85cdadfd694486590092a7b6f61f44c with delete_type TABLET_DATA_DELETED (replaced by tablet 9aadc67058554e87bb58d14b911333c4) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:01.614404 849 consensus_peers.cc:487] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:01.614451 743 tablet_service.cc:1515] Processing DeleteTablet for tablet d85cdadfd694486590092a7b6f61f44c with delete_type TABLET_DATA_DELETED (replaced by tablet 9aadc67058554e87bb58d14b911333c4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.618320 1442 raft_consensus.cc:491] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.618458 1442 raft_consensus.cc:513] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.618876 1166 tablet_bootstrap.cc:654] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.619107 1442 leader_election.cc:290] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:01.620064 851 leader_election.cc:343] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 13620f6ca65342b4ad7f1ba4b437bde3
W20250625 13:52:01.620939 849 leader_election.cc:343] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 13620f6ca65342b4ad7f1ba4b437bde3
I20250625 13:52:01.621063 849 leader_election.cc:304] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:01.621449 1442 raft_consensus.cc:2747] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:01.627192 1168 tablet_replica.cc:331] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:01.628544 1168 raft_consensus.cc:2241] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.628799 1168 raft_consensus.cc:2270] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.628741 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7448453a4e8c44ab8ad28eef52b99d5e" } new_table_name: "default.Table_938b272613694e4bb9b163d19e37053a" modify_external_catalogs: true
I20250625 13:52:01.631875 588 master_service.cc:948] ReplaceTablet: received request to replace tablet a29708fb688a4a65b84ae2564a040b4c from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:01.632109 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.635557 1168 ts_tablet_manager.cc:1905] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.628340 1171 tablet_bootstrap.cc:654] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.640640 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4ae5f1d414794474b2178b431cbc1d44
I20250625 13:52:01.641250 592 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:54762 (request call id 645) took 1183 ms. Trace:
I20250625 13:52:01.641479 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ae5f1d414794474b2178b431cbc1d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.641382 592 rpcz_store.cc:276] 0625 13:52:00.458052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.458119 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:00.458576 (+ 457us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 13:52:00.458738 (+ 162us) catalog_manager.cc:2790] Modifying in-memory table state
0625 13:52:00.458795 (+ 57us) catalog_manager.cc:2799] Locking tablets
0625 13:52:00.907514 (+448719us) catalog_manager.cc:2813] Removing table and tablets from system table
0625 13:52:01.640413 (+732899us) catalog_manager.cc:2831] Removing table from by-name map
0625 13:52:01.640500 (+ 87us) catalog_manager.cc:2847] Committing in-memory state
0625 13:52:01.640577 (+ 77us) catalog_manager.cc:2851] Aborting table tasks
0625 13:52:01.641196 (+ 619us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:52:01.616452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 13:52:01.616571 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0625 13:52:01.616577 (+ 6us) write_op.cc:435] Acquired schema lock
0625 13:52:01.616579 (+ 2us) tablet.cc:660] Decoding operations
0625 13:52:01.616703 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0625 13:52:01.616715 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0625 13:52:01.616720 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0625 13:52:01.617356 (+ 636us) tablet.cc:699] Row locks acquired
0625 13:52:01.617361 (+ 5us) write_op.cc:260] PREPARE: finished
0625 13:52:01.617398 (+ 37us) write_op.cc:270] Start()
0625 13:52:01.617432 (+ 34us) write_op.cc:276] Timestamp: P: 1750859521617379 usec, L: 0
0625 13:52:01.617436 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 13:52:01.634070 (+ 16634us) log.cc:844] Serialized 756 byte log entry
0625 13:52:01.634279 (+ 209us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x616000040f54. stack: 00007f0a6e369336 00007f0a6e3690f0 00007f0a6d35f3ad 0000000000971004 0000000000970fcd 000000000097099a 00007f0a79d9e762 00007f0a7b145855 00007f0a7b144379 00007f0a7b14c7a0 00007f0a7b14c5ad 0000000000a7573c 00007f0a6e3ae9df 00007f0a6e3b1cab 00007f0a6e3b1b28 0000000000a7573c
0625 13:52:01.638805 (+ 4526us) op_driver.cc:464] REPLICATION: finished
0625 13:52:01.639972 (+ 1167us) write_op.cc:301] APPLY: starting
0625 13:52:01.639988 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0625 13:52:01.640034 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0625 13:52:01.640036 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 13:52:01.640112 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 13:52:01.640145 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 13:52:01.640153 (+ 8us) write_op.cc:312] APPLY: finished
0625 13:52:01.640235 (+ 82us) log.cc:844] Serialized 29 byte log entry
0625 13:52:01.640308 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:52:01.640327 (+ 19us) write_op.cc:454] Released schema lock
0625 13:52:01.640334 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1090,"num_ops":2,"prepare.queue_time_us":708340,"prepare.run_cpu_time_us":1668,"prepare.run_wall_time_us":17886,"raft.queue_time_us":805,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":289,"replication_time_us":21341,"spinlock_wait_cycles":37758208}]]}
I20250625 13:52:01.642105 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ae5f1d414794474b2178b431cbc1d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.643668 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ae5f1d414794474b2178b431cbc1d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:01.647610 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a289b9e3fac4fd8bacc22c235f7010c consensus state has no leader
I20250625 13:52:01.654546 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7c6963b0fa6a4bce977904672ec5ae28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:01.655175 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet d9bb7ebc40a14ecaa0c692a8e1117db3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.655273 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 583ee2f2c66841939c93104efc36facb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:53 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.655735 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 02f2a2b595bc43309a29b3f1bc5d3766 with delete_type TABLET_DATA_DELETED (replaced by tablet e3939ef22ea344328721dfba67d933d1) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:01.657291 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c6963b0fa6a4bce977904672ec5ae28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.656467 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0ff1884a92fe4a4490d4a3f5594fa263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.657763 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ff1884a92fe4a4490d4a3f5594fa263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.659399 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 799f2edd960d45c1ace9195283a042e2 with delete_type TABLET_DATA_DELETED (replaced by tablet c7c3f0ca6fa5482db7f57143911c838f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.661301 1710 ts_tablet_manager.cc:1918] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:01.661464 1710 log.cc:1199] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d524089e6ab044ec961cfc9f587e8f86
I20250625 13:52:01.661921 1710 ts_tablet_manager.cc:1939] T d524089e6ab044ec961cfc9f587e8f86 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:01.662500 1166 tablet_bootstrap.cc:492] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.668576 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet d524089e6ab044ec961cfc9f587e8f86 (table default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]) successfully deleted
I20250625 13:52:01.671614 1166 ts_tablet_manager.cc:1397] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.001s
I20250625 13:52:01.671808 1171 tablet_bootstrap.cc:492] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.672030 1171 ts_tablet_manager.cc:1397] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20250625 13:52:01.672134 1170 tablet_bootstrap.cc:492] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.672276 1170 ts_tablet_manager.cc:1397] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.017s
I20250625 13:52:01.672523 1166 raft_consensus.cc:357] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.672729 1166 raft_consensus.cc:383] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.672827 1166 raft_consensus.cc:738] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.673058 1170 raft_consensus.cc:357] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.673228 1170 raft_consensus.cc:383] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.673311 1170 raft_consensus.cc:738] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.673137 1171 raft_consensus.cc:357] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.673523 1170 consensus_queue.cc:260] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.673725 1171 raft_consensus.cc:383] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.673849 1171 raft_consensus.cc:738] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.674028 1170 ts_tablet_manager.cc:1428] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:01.674189 1171 consensus_queue.cc:260] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.674387 1170 tablet_bootstrap.cc:492] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.675158 1165 ts_tablet_manager.cc:1918] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.675268 1165 log.cc:1199] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d0c5bea3b6214fa8af2792c5e9c411d9
I20250625 13:52:01.675675 1165 ts_tablet_manager.cc:1939] T d0c5bea3b6214fa8af2792c5e9c411d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:01.676442 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 487) took 1408 ms. Trace:
I20250625 13:52:01.676620 1170 tablet_bootstrap.cc:654] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.676685 1442 raft_consensus.cc:491] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.676813 1442 raft_consensus.cc:513] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.677439 1442 leader_election.cc:290] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:01.678531 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f73f420a4d7a438eadde3d6af64f0a11" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:01.679046 849 leader_election.cc:343] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.679705 1171 ts_tablet_manager.cc:1428] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
W20250625 13:52:01.680302 851 leader_election.cc:343] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f73f420a4d7a438eadde3d6af64f0a11
I20250625 13:52:01.680429 851 leader_election.cc:304] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:01.680976 1510 raft_consensus.cc:2747] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.681281 1171 tablet_bootstrap.cc:492] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:01.682037 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 063a793e0d7747668d2728fdc2160b5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 063a793e0d7747668d2728fdc2160b5e already in progress: creating tablet
I20250625 13:52:01.684227 1171 tablet_bootstrap.cc:654] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:01.684873 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4d2d6f05348e43339e7167a9cddca2ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2d6f05348e43339e7167a9cddca2ba already in progress: creating tablet
W20250625 13:52:01.686807 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ae4a105e80c1460cbb21e1bdb9b356b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae4a105e80c1460cbb21e1bdb9b356b1 already in progress: creating tablet
I20250625 13:52:01.686978 1166 consensus_queue.cc:260] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.687680 1166 ts_tablet_manager.cc:1428] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250625 13:52:01.688915 1168 ts_tablet_manager.cc:1918] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:01.689419 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d85cdadfd694486590092a7b6f61f44c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d85cdadfd694486590092a7b6f61f44c already in progress: creating tablet
I20250625 13:52:01.690146 1168 log.cc:1199] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/7fe053e294fe4aceb24919e9e2bddf65
W20250625 13:52:01.690606 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4ae5f1d414794474b2178b431cbc1d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ae5f1d414794474b2178b431cbc1d44 already in progress: creating tablet
I20250625 13:52:01.690938 1710 tablet_replica.cc:331] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:01.691197 1710 raft_consensus.cc:2241] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.691509 1710 raft_consensus.cc:2270] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.691656 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 17b172c7631048ceb97ccbea896f7f5b
I20250625 13:52:01.692490 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 17b172c7631048ceb97ccbea896f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.694324 1710 ts_tablet_manager.cc:1905] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.695856 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 17b172c7631048ceb97ccbea896f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.696372 1170 tablet_bootstrap.cc:492] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.696535 1170 ts_tablet_manager.cc:1397] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.001s
I20250625 13:52:01.693557 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 17b172c7631048ceb97ccbea896f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.697294 1170 raft_consensus.cc:357] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.697469 1170 raft_consensus.cc:383] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.697547 1170 raft_consensus.cc:738] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.697804 1170 consensus_queue.cc:260] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.698264 1168 ts_tablet_manager.cc:1939] T 7fe053e294fe4aceb24919e9e2bddf65 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:01.698354 1170 ts_tablet_manager.cc:1428] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:01.699615 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 208) took 6186 ms. Trace:
I20250625 13:52:01.700321 1170 tablet_bootstrap.cc:492] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.700480 1168 rpcz_store.cc:276] 0625 13:51:55.513019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.513062 (+ 43us) service_pool.cc:225] Handling call
0625 13:52:01.627100 (+6114038us) ts_tablet_manager.cc:1145] Deleting tablet 7fe053e294fe4aceb24919e9e2bddf65
0625 13:52:01.627111 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.669140 (+ 42029us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.688756 (+ 19616us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.699565 (+ 10809us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6113860}
I20250625 13:52:01.701169 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 7fe053e294fe4aceb24919e9e2bddf65 (table default.Table_d9dd2f12f8bb4f6cba435c892dac832e [id=98f8967275434f48a948def71ae1d693]) successfully deleted
I20250625 13:52:01.701961 1170 tablet_bootstrap.cc:654] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.702293 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 209) took 6139 ms. Trace:
I20250625 13:52:01.702479 1168 rpcz_store.cc:276] 0625 13:51:55.562686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.563113 (+ 427us) service_pool.cc:225] Handling call
0625 13:52:01.702143 (+6139030us) ts_tablet_manager.cc:1145] Deleting tablet 9a31f10c8ca745b7ad635c622a265450
0625 13:52:01.702152 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.702256 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6138776}
I20250625 13:52:01.704207 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 210) took 6116 ms. Trace:
I20250625 13:52:01.707667 1168 rpcz_store.cc:276] 0625 13:51:55.587856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.587912 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:01.704064 (+6116152us) ts_tablet_manager.cc:1145] Deleting tablet 8b7062c34fd143bf86b743055a27db8c
0625 13:52:01.704075 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.704177 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6072922}
W20250625 13:52:01.708621 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8b7062c34fd143bf86b743055a27db8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b7062c34fd143bf86b743055a27db8c already in progress: creating tablet
I20250625 13:52:01.709129 1168 tablet_replica.cc:331] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:01.710224 1168 raft_consensus.cc:2241] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.710664 1168 raft_consensus.cc:2270] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.711697 1168 ts_tablet_manager.cc:1905] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.712772 1166 tablet_bootstrap.cc:492] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.713618 1171 tablet_bootstrap.cc:492] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.713862 1171 ts_tablet_manager.cc:1397] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.005s
I20250625 13:52:01.714999 1171 raft_consensus.cc:357] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.715255 1171 raft_consensus.cc:383] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.715415 1171 raft_consensus.cc:738] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.715690 1171 consensus_queue.cc:260] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.716399 1166 tablet_bootstrap.cc:654] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.717486 1165 rpcz_store.cc:276] 0625 13:52:00.267482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.267536 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:01.537218 (+1269682us) ts_tablet_manager.cc:1145] Deleting tablet d0c5bea3b6214fa8af2792c5e9c411d9
0625 13:52:01.537227 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.601544 (+ 64317us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.675017 (+ 73473us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.676405 (+ 1388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1121699}
I20250625 13:52:01.718027 1695 raft_consensus.cc:491] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.718051 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d0c5bea3b6214fa8af2792c5e9c411d9 (table default.Table_77082a1ec23a48adb1f251559dc90002 [id=d482f250757e4eaaa5a5a4daa115bd34]) successfully deleted
I20250625 13:52:01.718222 1695 raft_consensus.cc:513] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.718791 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 488) took 1434 ms. Trace:
I20250625 13:52:01.718901 1165 rpcz_store.cc:276] 0625 13:52:00.284705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.284762 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:01.718636 (+1433874us) ts_tablet_manager.cc:1145] Deleting tablet ed19663d21d84c9e90d4f4dda055420d
0625 13:52:01.718646 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.718758 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1303071}
I20250625 13:52:01.719096 1170 tablet_bootstrap.cc:492] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.719260 1170 ts_tablet_manager.cc:1397] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.007s
W20250625 13:52:01.719363 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ed19663d21d84c9e90d4f4dda055420d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed19663d21d84c9e90d4f4dda055420d already in progress: creating tablet
I20250625 13:52:01.719116 1695 leader_election.cc:290] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.720093 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41a7862601f04fd7a938f7e5d5a4c0f9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:01.720155 1170 raft_consensus.cc:357] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.720244 1171 ts_tablet_manager.cc:1428] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:52:01.720309 1170 raft_consensus.cc:383] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.720391 1170 raft_consensus.cc:738] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.720606 1171 tablet_bootstrap.cc:492] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:01.720654 1170 consensus_queue.cc:260] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.721195 1170 ts_tablet_manager.cc:1428] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:01.721482 1170 tablet_bootstrap.cc:492] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:01.723263 985 leader_election.cc:343] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 41a7862601f04fd7a938f7e5d5a4c0f9
I20250625 13:52:01.723423 985 leader_election.cc:304] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.723803 1170 tablet_bootstrap.cc:654] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.724797 1695 raft_consensus.cc:2747] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.726424 1171 tablet_bootstrap.cc:654] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.727806 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 489) took 1362 ms. Trace:
I20250625 13:52:01.728332 1710 ts_tablet_manager.cc:1918] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.728500 1710 log.cc:1199] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d9bb7ebc40a14ecaa0c692a8e1117db3
I20250625 13:52:01.728708 1165 rpcz_store.cc:276] 0625 13:52:00.365518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.365582 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:01.727652 (+1362070us) ts_tablet_manager.cc:1145] Deleting tablet efc9536a716f4b9f9fc4af1637b3cd80
0625 13:52:01.727663 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.727768 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311977}
I20250625 13:52:01.728883 1710 ts_tablet_manager.cc:1939] T d9bb7ebc40a14ecaa0c692a8e1117db3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:01.729583 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet efc9536a716f4b9f9fc4af1637b3cd80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efc9536a716f4b9f9fc4af1637b3cd80 already in progress: creating tablet
I20250625 13:52:01.729933 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet d9bb7ebc40a14ecaa0c692a8e1117db3 (table default.Table_40de6f83d3af49f1a0bc517c4d0d875e [id=6e50ba5e257842e080a70caa1679952d]) successfully deleted
I20250625 13:52:01.730122 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 485) took 1600 ms. Trace:
I20250625 13:52:01.730592 1165 rpcz_store.cc:276] 0625 13:52:00.129719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.398035 (+268316us) service_pool.cc:225] Handling call
0625 13:52:01.729984 (+1331949us) ts_tablet_manager.cc:1145] Deleting tablet ff65e23c44624b2cbf92c2dc67eaed09
0625 13:52:01.729993 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.730089 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280022}
W20250625 13:52:01.734942 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f06030f45284c5a9929a4c4513004ba has no consensus state
I20250625 13:52:01.735675 1710 tablet_replica.cc:331] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:01.735901 1710 raft_consensus.cc:2241] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.736104 1710 raft_consensus.cc:2270] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.736608 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a29708fb688a4a65b84ae2564a040b4c
I20250625 13:52:01.737074 588 catalog_manager.cc:6652] ReplaceTablet: tablet a29708fb688a4a65b84ae2564a040b4c deleted and replaced by tablet 2c8cf5d6cd624a1797486c69bd2098fd
I20250625 13:52:01.737330 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 553af7c7326240838f37e3861d4e3357 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:01.737960 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet a29708fb688a4a65b84ae2564a040b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 2c8cf5d6cd624a1797486c69bd2098fd) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.738497 1170 tablet_bootstrap.cc:492] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.738544 886 tablet_service.cc:1515] Processing DeleteTablet for tablet a29708fb688a4a65b84ae2564a040b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 2c8cf5d6cd624a1797486c69bd2098fd) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.738653 1170 ts_tablet_manager.cc:1397] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.001s
I20250625 13:52:01.739449 1170 raft_consensus.cc:357] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.739549 1710 ts_tablet_manager.cc:1905] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.739641 1170 raft_consensus.cc:383] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.739717 1170 raft_consensus.cc:738] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.739962 1170 consensus_queue.cc:260] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.740403 1170 ts_tablet_manager.cc:1428] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:01.740736 1170 tablet_bootstrap.cc:492] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.742362 1170 tablet_bootstrap.cc:654] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.748853 1170 tablet_bootstrap.cc:492] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.749009 1170 ts_tablet_manager.cc:1397] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20250625 13:52:01.749737 1170 raft_consensus.cc:357] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.749948 1170 raft_consensus.cc:383] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.750027 1170 raft_consensus.cc:738] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.750227 1170 consensus_queue.cc:260] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.750743 1170 ts_tablet_manager.cc:1428] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:01.751046 1170 tablet_bootstrap.cc:492] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.751283 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 2404e712e18146b68757bd77362c1358 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:01.751888 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 41a7862601f04fd7a938f7e5d5a4c0f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.752321 1171 tablet_bootstrap.cc:492] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.752518 1171 ts_tablet_manager.cc:1397] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20250625 13:52:01.752723 1170 tablet_bootstrap.cc:654] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.753691 1171 raft_consensus.cc:357] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.753926 1171 raft_consensus.cc:383] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.754022 1171 raft_consensus.cc:738] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.754277 1171 consensus_queue.cc:260] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.754993 1171 ts_tablet_manager.cc:1428] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:01.755383 1171 tablet_bootstrap.cc:492] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:01.755581 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7e64f15192894423be9dcd6d12afcc67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.756153 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e64f15192894423be9dcd6d12afcc67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.757120 1171 tablet_bootstrap.cc:654] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.757598 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c7ffa1aa92d44d7a29c530d6b9279e6" } new_table_name: "default.Table_034f6cf597cd469fba12687016eff7d1" modify_external_catalogs: true
I20250625 13:52:01.758388 1168 ts_tablet_manager.cc:1918] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.758518 1168 log.cc:1199] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/1a7b72c2c464421eacfcf64008c3aea8
I20250625 13:52:01.758888 1168 ts_tablet_manager.cc:1939] T 1a7b72c2c464421eacfcf64008c3aea8 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:01.759155 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a07c15abd0224e7d80d182ea2c891993" } new_table_name: "default.Table_295dd4f121234e659090b8e4cf70faee" modify_external_catalogs: true
I20250625 13:52:01.761174 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.762136 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 211) took 6101 ms. Trace:
I20250625 13:52:01.762229 1168 rpcz_store.cc:276] 0625 13:51:55.660416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.660467 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:01.709047 (+6048580us) ts_tablet_manager.cc:1145] Deleting tablet 1a7b72c2c464421eacfcf64008c3aea8
0625 13:52:01.709061 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.734180 (+ 25119us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.758257 (+ 24077us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.762071 (+ 3814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6048345}
I20250625 13:52:01.762526 1170 tablet_bootstrap.cc:492] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.762693 1170 ts_tablet_manager.cc:1397] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250625 13:52:01.762732 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d8fbb4df3d0745b6a9671d0dbfb1c789"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:01.762950 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 1a7b72c2c464421eacfcf64008c3aea8 (table default.Table_3599147b4b6b4aa3bcaa6de4475ae800 [id=cd3d67e4b72d4d919344482f7b10ed10]) successfully deleted
W20250625 13:52:01.763273 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8fbb4df3d0745b6a9671d0dbfb1c789 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.763466 1170 raft_consensus.cc:357] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:01.763680 1170 raft_consensus.cc:383] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.763756 1170 raft_consensus.cc:738] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.764019 1170 consensus_queue.cc:260] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:01.764492 1170 ts_tablet_manager.cc:1428] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:01.764761 1170 tablet_bootstrap.cc:492] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.764879 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 214) took 6052 ms. Trace:
I20250625 13:52:01.764978 1168 rpcz_store.cc:276] 0625 13:51:55.712687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.713592 (+ 905us) service_pool.cc:225] Handling call
0625 13:52:01.764704 (+6051112us) ts_tablet_manager.cc:1145] Deleting tablet 8fdca99b734448118e0c42227c78e58f
0625 13:52:01.764713 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.764827 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6045784}
W20250625 13:52:01.765427 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8fdca99b734448118e0c42227c78e58f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fdca99b734448118e0c42227c78e58f already in progress: creating tablet
I20250625 13:52:01.765967 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 216) took 6053 ms. Trace:
I20250625 13:52:01.766438 1170 tablet_bootstrap.cc:654] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.766485 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet f73f420a4d7a438eadde3d6af64f0a11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.766930 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6be7189f8f45ccb329ef5d2fd8da8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.766722 1168 rpcz_store.cc:276] 0625 13:51:55.712905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.713665 (+ 760us) service_pool.cc:225] Handling call
0625 13:52:01.765794 (+6052129us) ts_tablet_manager.cc:1145] Deleting tablet 2bbd509e3dfb45d98314c6afe37677f6
0625 13:52:01.765806 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.765921 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6045907}
I20250625 13:52:01.767381 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 93f449c80b604a6498f6fd1d1ba4770a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.767773 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ba63184ca2b4badafcf08309ef75164 with delete_type TABLET_DATA_DELETED (replaced by tablet 829221b8a79f460d90dd24be6a2d4c43) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.768337 1168 tablet_replica.cc:331] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:01.769071 1168 raft_consensus.cc:2241] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.772069 886 tablet_service.cc:1515] Processing DeleteTablet for tablet c911d9ecbded417daf70e85c7cbcd52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.772868 1168 raft_consensus.cc:2270] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.773572 1168 ts_tablet_manager.cc:1905] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.775050 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 85c72f9ae92a4537979fb479eb4467c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.775550 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0f82cde67a844d2dadb19c95b952db96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.776171 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f82cde67a844d2dadb19c95b952db96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.776413 1171 tablet_bootstrap.cc:492] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.776582 1171 ts_tablet_manager.cc:1397] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20250625 13:52:01.777454 1171 raft_consensus.cc:357] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:01.777688 1171 raft_consensus.cc:383] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.777828 1171 raft_consensus.cc:738] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.778136 1171 consensus_queue.cc:260] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:01.781610 1171 ts_tablet_manager.cc:1428] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250625 13:52:01.781795 1170 tablet_bootstrap.cc:492] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.781960 1170 ts_tablet_manager.cc:1397] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20250625 13:52:01.781998 1171 tablet_bootstrap.cc:492] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:01.782742 1170 raft_consensus.cc:357] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.782945 1170 raft_consensus.cc:383] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.783046 1170 raft_consensus.cc:738] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.783267 1170 consensus_queue.cc:260] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.783754 1170 ts_tablet_manager.cc:1428] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:01.783915 1710 ts_tablet_manager.cc:1918] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.784691 1171 tablet_bootstrap.cc:654] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.784049 1170 tablet_bootstrap.cc:492] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.785387 1710 log.cc:1199] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/583ee2f2c66841939c93104efc36facb
I20250625 13:52:01.790035 1046 raft_consensus.cc:3058] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.790741 1170 tablet_bootstrap.cc:654] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.794137 1710 ts_tablet_manager.cc:1939] T 583ee2f2c66841939c93104efc36facb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:01.738193 743 tablet_service.cc:1515] Processing DeleteTablet for tablet a29708fb688a4a65b84ae2564a040b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 2c8cf5d6cd624a1797486c69bd2098fd) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.764961 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 46316eb725bc47e695af476c0b185504 with delete_type TABLET_DATA_DELETED (replaced by tablet b7d888e628a94f939ecf2571e60b1948) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.782778 1669 raft_consensus.cc:491] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.795640 1669 raft_consensus.cc:513] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:01.795771 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ff65e23c44624b2cbf92c2dc67eaed09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff65e23c44624b2cbf92c2dc67eaed09 already in progress: creating tablet
I20250625 13:52:01.796288 1669 leader_election.cc:290] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.796590 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 490) took 1207 ms. Trace:
I20250625 13:52:01.796722 1165 rpcz_store.cc:276] 0625 13:52:00.588839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.589072 (+ 233us) service_pool.cc:225] Handling call
0625 13:52:01.796410 (+1207338us) ts_tablet_manager.cc:1145] Deleting tablet afa1c198dd6541809843244c7cc18194
0625 13:52:01.796423 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.796542 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1169837}
I20250625 13:52:01.797062 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1acc41a93f6433588727811903921fe" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:01.797312 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet afa1c198dd6541809843244c7cc18194 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afa1c198dd6541809843244c7cc18194 already in progress: creating tablet
W20250625 13:52:01.797576 711 leader_election.cc:343] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.797763 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 491) took 1169 ms. Trace:
I20250625 13:52:01.797904 1165 rpcz_store.cc:276] 0625 13:52:00.628652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.631882 (+ 3230us) service_pool.cc:225] Handling call
0625 13:52:01.797629 (+1165747us) ts_tablet_manager.cc:1145] Deleting tablet c605d5b64c164c1594b0a40cfa2c4cf6
0625 13:52:01.797639 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.797727 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159961}
W20250625 13:52:01.798421 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c605d5b64c164c1594b0a40cfa2c4cf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c605d5b64c164c1594b0a40cfa2c4cf6 already in progress: creating tablet
I20250625 13:52:01.798863 1165 tablet_replica.cc:331] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:01.799157 1165 raft_consensus.cc:2241] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.800072 1165 raft_consensus.cc:2270] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.801023 1165 ts_tablet_manager.cc:1905] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.801183 1170 tablet_bootstrap.cc:492] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.801358 1170 ts_tablet_manager.cc:1397] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20250625 13:52:01.802029 1170 raft_consensus.cc:357] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.802204 1170 raft_consensus.cc:383] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.802287 1170 raft_consensus.cc:738] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.802526 1170 consensus_queue.cc:260] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.803048 1170 ts_tablet_manager.cc:1428] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:01.803287 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1acc41a93f6433588727811903921fe" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:01.803534 1170 tablet_bootstrap.cc:492] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:01.804196 711 leader_election.cc:343] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.804320 711 leader_election.cc:304] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.804890 1669 raft_consensus.cc:2747] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.805235 1170 tablet_bootstrap.cc:654] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.805368 1710 tablet_replica.cc:331] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:01.805567 1710 raft_consensus.cc:2241] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.805718 1710 raft_consensus.cc:2270] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.807633 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 583ee2f2c66841939c93104efc36facb (table default.Table_5964baca5b4a4701bf5a1b86d494b4e0 [id=c12c680f4ece4270ba65d0e2dc275254]) successfully deleted
I20250625 13:52:01.812136 1166 tablet_bootstrap.cc:492] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.812364 1166 ts_tablet_manager.cc:1397] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.100s user 0.000s sys 0.004s
I20250625 13:52:01.813839 1166 raft_consensus.cc:357] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:01.814124 1166 raft_consensus.cc:383] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.814246 1166 raft_consensus.cc:738] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.814558 1166 consensus_queue.cc:260] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:01.815136 1166 ts_tablet_manager.cc:1428] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:01.818035 591 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 50786a4c25274ffb8f9910c5dafe3933 (table default.Table_ebb1cd6aa381410a82cac38e7197353c [id=d7cbe338b2554a6e99b07325632aaa33]). Expected version 3 got 0
I20250625 13:52:01.818809 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_09dd798f73ca46cf9edc4fd651169efd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:01.819553 1604 raft_consensus.cc:491] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.819595 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_db99293d790941fcbd2bbb8a9ed7967c" } new_table_name: "default.Table_5df71374672448ce89f07be4d9f97e41" modify_external_catalogs: true
I20250625 13:52:01.819710 1604 raft_consensus.cc:513] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.820395 1604 leader_election.cc:290] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:01.821179 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e58fbec94e4ff9b7ac9830b576fe7f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:01.821398 1050 raft_consensus.cc:2466] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:01.821847 849 leader_election.cc:304] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:01.821916 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09dd798f73ca46cf9edc4fd651169efd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.824239 1518 raft_consensus.cc:491] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.824399 1518 raft_consensus.cc:513] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.825049 1518 leader_election.cc:290] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.825603 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ba63184ca2b4badafcf08309ef75164" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:01.825688 1170 tablet_bootstrap.cc:492] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.825814 1170 ts_tablet_manager.cc:1397] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.001s
I20250625 13:52:01.825798 913 raft_consensus.cc:2466] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:01.826568 983 leader_election.cc:304] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:01.826534 1170 raft_consensus.cc:357] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.826725 1170 raft_consensus.cc:383] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.826821 1170 raft_consensus.cc:738] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:01.827903 985 leader_election.cc:343] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9ba63184ca2b4badafcf08309ef75164
I20250625 13:52:01.827037 1170 consensus_queue.cc:260] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.828279 1518 raft_consensus.cc:2802] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:01.828367 1518 raft_consensus.cc:491] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:01.828466 1518 raft_consensus.cc:3058] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.828665 1170 ts_tablet_manager.cc:1428] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:01.830091 1170 tablet_bootstrap.cc:492] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.831259 1510 raft_consensus.cc:2802] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:01.831393 1510 raft_consensus.cc:491] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:01.831476 1510 raft_consensus.cc:3058] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.832495 1170 tablet_bootstrap.cc:654] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.834195 1510 raft_consensus.cc:513] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.835896 1510 leader_election.cc:290] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:01.837724 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e58fbec94e4ff9b7ac9830b576fe7f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:01.837934 1050 raft_consensus.cc:3058] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.821159 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e58fbec94e4ff9b7ac9830b576fe7f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:01.841955 776 raft_consensus.cc:2466] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:01.842132 1166 tablet_bootstrap.cc:492] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.835390 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e58fbec94e4ff9b7ac9830b576fe7f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:01.843782 777 raft_consensus.cc:3058] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.844882 1166 tablet_bootstrap.cc:654] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.846652 777 raft_consensus.cc:2466] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:01.848526 851 leader_election.cc:304] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:01.849151 1510 raft_consensus.cc:2802] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:01.849799 1171 tablet_bootstrap.cc:492] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:01.850382 1518 raft_consensus.cc:513] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.853246 1050 raft_consensus.cc:2466] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:01.854012 1710 ts_tablet_manager.cc:1905] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.849890 1510 raft_consensus.cc:695] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:01.854399 1171 ts_tablet_manager.cc:1397] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.002s
I20250625 13:52:01.855331 1518 leader_election.cc:290] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:01.855489 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc75a0a8421b457089cc707d9eb6b76f consensus state has no leader
I20250625 13:52:01.857244 1170 tablet_bootstrap.cc:492] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:01.857242 985 leader_election.cc:343] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9ba63184ca2b4badafcf08309ef75164
I20250625 13:52:01.857560 1170 ts_tablet_manager.cc:1397] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.010s
I20250625 13:52:01.854458 1510 consensus_queue.cc:237] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.858676 1171 raft_consensus.cc:357] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.858909 1171 raft_consensus.cc:383] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.858984 1171 raft_consensus.cc:738] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.855898 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ba63184ca2b4badafcf08309ef75164" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:01.859200 913 raft_consensus.cc:3058] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:01.859778 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b14d00c93a964edd815122e3723f89e7 consensus state has no leader
W20250625 13:52:01.861140 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af704cdf44ef42f7b5ea10b199d0f017 consensus state has no leader
I20250625 13:52:01.862015 913 raft_consensus.cc:2466] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:01.862709 983 leader_election.cc:304] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:01.864727 1171 consensus_queue.cc:260] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.865783 1171 ts_tablet_manager.cc:1428] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 13:52:01.866335 1518 raft_consensus.cc:2802] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:01.866788 1518 raft_consensus.cc:695] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:01.867136 1518 consensus_queue.cc:237] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.867321 1604 raft_consensus.cc:491] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.867477 1604 raft_consensus.cc:513] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.868213 1604 leader_election.cc:290] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:01.869385 851 leader_election.cc:343] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1f6be7189f8f45ccb329ef5d2fd8da8d
I20250625 13:52:01.869783 1604 raft_consensus.cc:491] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.869940 1604 raft_consensus.cc:513] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.870612 1604 leader_election.cc:290] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:01.871775 851 leader_election.cc:343] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: c911d9ecbded417daf70e85c7cbcd52e
I20250625 13:52:01.873358 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c911d9ecbded417daf70e85c7cbcd52e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:01.873437 1170 raft_consensus.cc:357] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.874363 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6be7189f8f45ccb329ef5d2fd8da8d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:01.874729 1050 raft_consensus.cc:2466] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:01.874722 1170 raft_consensus.cc:383] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.875286 849 leader_election.cc:304] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:01.875089 1170 raft_consensus.cc:738] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.876400 1170 consensus_queue.cc:260] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.879632 1049 raft_consensus.cc:2466] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:01.880697 1170 ts_tablet_manager.cc:1428] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20250625 13:52:01.881916 1170 tablet_bootstrap.cc:492] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.886304 1669 raft_consensus.cc:491] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.886484 1669 raft_consensus.cc:513] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.887235 1669 leader_election.cc:290] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.887776 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "055f88e118454e32baa578cae9aceb0c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:01.888485 1170 tablet_bootstrap.cc:654] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:01.888573 711 leader_election.cc:343] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.888893 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "055f88e118454e32baa578cae9aceb0c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:01.889390 711 leader_election.cc:343] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.889523 711 leader_election.cc:304] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.889796 1669 raft_consensus.cc:2747] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.897915 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e58fbec94e4ff9b7ac9830b576fe7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.901240 1170 tablet_bootstrap.cc:492] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:01.901407 1170 ts_tablet_manager.cc:1397] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.002s
I20250625 13:52:01.904170 1170 raft_consensus.cc:357] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.904381 1170 raft_consensus.cc:383] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.904472 1170 raft_consensus.cc:738] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.904695 1170 consensus_queue.cc:260] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.904985 1166 tablet_bootstrap.cc:492] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:01.905177 1170 ts_tablet_manager.cc:1428] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:01.905254 1166 ts_tablet_manager.cc:1397] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.002s
I20250625 13:52:01.905474 1170 tablet_bootstrap.cc:492] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:01.906002 1166 raft_consensus.cc:357] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.906261 1166 raft_consensus.cc:383] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:01.906383 1166 raft_consensus.cc:738] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:01.906672 1166 consensus_queue.cc:260] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.907222 1166 ts_tablet_manager.cc:1428] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:01.908144 1170 tablet_bootstrap.cc:654] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.912251 1747 raft_consensus.cc:2802] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:01.912413 1747 raft_consensus.cc:491] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:01.912581 1747 raft_consensus.cc:3058] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.916918 849 leader_election.cc:304] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:01.917438 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20d0eb2c546641379a1f7ff2c77427d1" } modify_external_catalogs: true
I20250625 13:52:01.917682 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20d0eb2c546641379a1f7ff2c77427d1" } modify_external_catalogs: true
I20250625 13:52:01.917894 1171 tablet_bootstrap.cc:492] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:01.918474 1510 raft_consensus.cc:2802] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:01.918645 1510 raft_consensus.cc:491] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:01.918759 1510 raft_consensus.cc:3058] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.919040 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b2b0087eeea54823809b847648141d81" } modify_external_catalogs: true
I20250625 13:52:01.919823 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b2b0087eeea54823809b847648141d81" } modify_external_catalogs: true
I20250625 13:52:01.922240 1510 raft_consensus.cc:513] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.923570 1510 leader_election.cc:290] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:01.923571 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_407b8d22e01548a4aadf917bd73fa8f6" } new_table_name: "default.Table_585826da245840dab264effeab34e150" modify_external_catalogs: true
I20250625 13:52:01.924363 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c911d9ecbded417daf70e85c7cbcd52e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:01.924566 1049 raft_consensus.cc:3058] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.926718 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2e709eb4a94545c58de6814fff368a89" } modify_external_catalogs: true
I20250625 13:52:01.926965 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2e709eb4a94545c58de6814fff368a89" } modify_external_catalogs: true
I20250625 13:52:01.927423 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet c911d9ecbded417daf70e85c7cbcd52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.927557 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6820dad35d484763a041822f9b7bd57f" } modify_external_catalogs: true
I20250625 13:52:01.927657 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c5bea3b6214fa8af2792c5e9c411d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.927740 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6820dad35d484763a041822f9b7bd57f" } modify_external_catalogs: true
I20250625 13:52:01.930711 1049 raft_consensus.cc:2466] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:01.931288 849 leader_election.cc:304] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:01.932572 1604 raft_consensus.cc:2802] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:01.932780 1604 raft_consensus.cc:695] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:01.935046 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet cf1664afbab647b3baeed6391196bed9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.935129 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0ca3566f743046db9a91f2a4040f2e03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.935822 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ca3566f743046db9a91f2a4040f2e03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.934077 1604 consensus_queue.cc:237] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.940007 1171 tablet_bootstrap.cc:654] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.932755 1168 ts_tablet_manager.cc:1918] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:01.944064 1168 log.cc:1199] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/cbaa018753d8455199f3869323fb5582
I20250625 13:52:01.944090 1166 tablet_bootstrap.cc:492] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:01.944623 1168 ts_tablet_manager.cc:1939] T cbaa018753d8455199f3869323fb5582 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:01.945883 851 leader_election.cc:343] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: c911d9ecbded417daf70e85c7cbcd52e
I20250625 13:52:01.945937 1166 tablet_bootstrap.cc:654] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:01.946442 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 226) took 6211 ms. Trace:
I20250625 13:52:01.946583 1168 rpcz_store.cc:276] 0625 13:51:55.734512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.734565 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:01.768253 (+6033688us) ts_tablet_manager.cc:1145] Deleting tablet cbaa018753d8455199f3869323fb5582
0625 13:52:01.768261 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.807981 (+ 39720us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.919008 (+111027us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:01.946407 (+ 27399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6033485}
I20250625 13:52:01.949820 1710 ts_tablet_manager.cc:1918] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:01.949975 1710 log.cc:1199] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/02f2a2b595bc43309a29b3f1bc5d3766
I20250625 13:52:01.950384 1710 ts_tablet_manager.cc:1939] T 02f2a2b595bc43309a29b3f1bc5d3766 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:01.951432 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 228) took 6188 ms. Trace:
I20250625 13:52:01.951464 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 02f2a2b595bc43309a29b3f1bc5d3766 (table default.Table_630c47cb56984efbac74836e412efaa6 [id=8c054766f5424144a4edb39b192451cb]) successfully deleted
I20250625 13:52:01.951542 1168 rpcz_store.cc:276] 0625 13:51:55.763070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.763165 (+ 95us) service_pool.cc:225] Handling call
0625 13:52:01.951268 (+6188103us) ts_tablet_manager.cc:1145] Deleting tablet 58aa3c21191e4bea9c5395d6e32b8717
0625 13:52:01.951278 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.951401 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6187808}
W20250625 13:52:01.952994 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 58aa3c21191e4bea9c5395d6e32b8717 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58aa3c21191e4bea9c5395d6e32b8717 already in progress: creating tablet
I20250625 13:52:01.953823 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 218) took 6240 ms. Trace:
I20250625 13:52:01.954454 1168 rpcz_store.cc:276] 0625 13:51:55.713118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.713725 (+ 607us) service_pool.cc:225] Handling call
0625 13:52:01.953696 (+6239971us) ts_tablet_manager.cc:1145] Deleting tablet 8e29e7b55570483b80c6e7a1f227fa7b
0625 13:52:01.953705 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.953785 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6099682}
W20250625 13:52:01.955504 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8e29e7b55570483b80c6e7a1f227fa7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e29e7b55570483b80c6e7a1f227fa7b already in progress: creating tablet
I20250625 13:52:01.956203 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 224) took 6224 ms. Trace:
I20250625 13:52:01.956877 1168 rpcz_store.cc:276] 0625 13:51:55.731774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.732078 (+ 304us) service_pool.cc:225] Handling call
0625 13:52:01.956083 (+6224005us) ts_tablet_manager.cc:1145] Deleting tablet 637847cac1724bfeb629c4d4013396ca
0625 13:52:01.956092 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.956176 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6086308}
W20250625 13:52:01.957646 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 637847cac1724bfeb629c4d4013396ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 637847cac1724bfeb629c4d4013396ca already in progress: creating tablet
I20250625 13:52:01.958501 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 225) took 6226 ms. Trace:
I20250625 13:52:01.958925 1168 rpcz_store.cc:276] 0625 13:51:55.731920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.732212 (+ 292us) service_pool.cc:225] Handling call
0625 13:52:01.957986 (+6225774us) ts_tablet_manager.cc:1145] Deleting tablet c0a3ea18bab944749d45da9b3139f8bc
0625 13:52:01.957995 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.958469 (+ 474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6087128}
W20250625 13:52:01.959990 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c0a3ea18bab944749d45da9b3139f8bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0a3ea18bab944749d45da9b3139f8bc already in progress: creating tablet
I20250625 13:52:01.960460 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 227) took 6214 ms. Trace:
I20250625 13:52:01.961830 1168 rpcz_store.cc:276] 0625 13:51:55.745692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:55.745750 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:01.960340 (+6214590us) ts_tablet_manager.cc:1145] Deleting tablet 54fa042947da4e1b89018640b46b271a
0625 13:52:01.960349 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.960430 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6036848}
W20250625 13:52:01.962771 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 54fa042947da4e1b89018640b46b271a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54fa042947da4e1b89018640b46b271a already in progress: creating tablet
I20250625 13:52:01.963536 1168 tablet_replica.cc:331] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:01.966681 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 3273597f614d4d0a9f89fe8bfbe0a421
I20250625 13:52:01.967001 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:01.967032 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f89c8b7c36634729a135e0a425694693"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.967624 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f89c8b7c36634729a135e0a425694693 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:01.967773 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0998ad87fcc145999f6430e0de7b1304 consensus state has no leader
I20250625 13:52:01.967433 1168 raft_consensus.cc:2241] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.968107 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 11711546f0e14f9a9eabf003f49a69b7 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:01.968513 1168 raft_consensus.cc:2270] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.968871 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2fb3cd371d784e4c980b6f63c84d2f09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:01.969484 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fb3cd371d784e4c980b6f63c84d2f09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.970409 1168 ts_tablet_manager.cc:1905] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:01.974589 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0e3ef7f5b474466bbc19fd31c6b6d99c
I20250625 13:52:01.975180 589 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:54762 (request call id 676) took 1141 ms. Trace:
I20250625 13:52:01.975485 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.975553 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:01.975667 1710 tablet_replica.cc:331] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:01.975277 589 rpcz_store.cc:276] 0625 13:52:00.833431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.573700 (+740269us) service_pool.cc:225] Handling call
0625 13:52:01.574325 (+ 625us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 13:52:01.574467 (+ 142us) catalog_manager.cc:2790] Modifying in-memory table state
0625 13:52:01.574550 (+ 83us) catalog_manager.cc:2799] Locking tablets
0625 13:52:01.964825 (+390275us) catalog_manager.cc:2813] Removing table and tablets from system table
0625 13:52:01.974429 (+ 9604us) catalog_manager.cc:2831] Removing table from by-name map
0625 13:52:01.974499 (+ 70us) catalog_manager.cc:2847] Committing in-memory state
0625 13:52:01.974531 (+ 32us) catalog_manager.cc:2851] Aborting table tasks
0625 13:52:01.975129 (+ 598us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:52:01.970351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 13:52:01.970456 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0625 13:52:01.970461 (+ 5us) write_op.cc:435] Acquired schema lock
0625 13:52:01.970465 (+ 4us) tablet.cc:660] Decoding operations
0625 13:52:01.970597 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0625 13:52:01.970608 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0625 13:52:01.970613 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0625 13:52:01.970696 (+ 83us) tablet.cc:699] Row locks acquired
0625 13:52:01.970698 (+ 2us) write_op.cc:260] PREPARE: finished
0625 13:52:01.970735 (+ 37us) write_op.cc:270] Start()
0625 13:52:01.970766 (+ 31us) write_op.cc:276] Timestamp: P: 1750859521970680 usec, L: 0
0625 13:52:01.970772 (+ 6us) op_driver.cc:348] REPLICATION: starting
0625 13:52:01.970935 (+ 163us) log.cc:844] Serialized 725 byte log entry
0625 13:52:01.973788 (+ 2853us) op_driver.cc:464] REPLICATION: finished
0625 13:52:01.973979 (+ 191us) write_op.cc:301] APPLY: starting
0625 13:52:01.973992 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0625 13:52:01.974032 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0625 13:52:01.974054 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 13:52:01.974116 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 13:52:01.974145 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 13:52:01.974153 (+ 8us) write_op.cc:312] APPLY: finished
0625 13:52:01.974215 (+ 62us) log.cc:844] Serialized 29 byte log entry
0625 13:52:01.974281 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:52:01.974299 (+ 18us) write_op.cc:454] Released schema lock
0625 13:52:01.974305 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":383,"apply.run_wall_time_us":382,"num_ops":2,"prepare.queue_time_us":4416,"raft.queue_time_us":38,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":238,"replication_time_us":2988}]]}
I20250625 13:52:01.976691 1710 raft_consensus.cc:2241] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:01.977193 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.977510 1710 raft_consensus.cc:2270] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:01.978281 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f5d01b07d90441a79d2ccd6c8fa92610"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:01.978428 1710 ts_tablet_manager.cc:1905] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:01.978808 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5d01b07d90441a79d2ccd6c8fa92610 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:01.979802 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
I20250625 13:52:01.980273 1669 raft_consensus.cc:491] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:01.980463 1669 raft_consensus.cc:513] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.980523 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_550407055545488f84dc060ae1891f73" } new_table_name: "default.Table_def99fb8fc1f4cdd86436da682cdae13" modify_external_catalogs: true
I20250625 13:52:01.981287 1669 leader_election.cc:290] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:01.982160 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46316eb725bc47e695af476c0b185504" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:01.982966 711 leader_election.cc:343] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.982933 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46316eb725bc47e695af476c0b185504" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:01.983492 711 leader_election.cc:343] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:01.983611 711 leader_election.cc:304] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:01.984042 1669 raft_consensus.cc:2747] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:01.990861 1747 raft_consensus.cc:513] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:01.991067 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2404e712e18146b68757bd77362c1358
I20250625 13:52:01.991604 594 catalog_manager.cc:6652] ReplaceTablet: tablet 2404e712e18146b68757bd77362c1358 deleted and replaced by tablet 68831b9ad1c14f5e9c954cbd06094896
I20250625 13:52:01.992797 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 2404e712e18146b68757bd77362c1358 with delete_type TABLET_DATA_DELETED (replaced by tablet 68831b9ad1c14f5e9c954cbd06094896) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:01.993009 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 2404e712e18146b68757bd77362c1358 with delete_type TABLET_DATA_DELETED (replaced by tablet 68831b9ad1c14f5e9c954cbd06094896) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:01.993949 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 2404e712e18146b68757bd77362c1358 with delete_type TABLET_DATA_DELETED (replaced by tablet 68831b9ad1c14f5e9c954cbd06094896) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:01.995671 851 leader_election.cc:343] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1f6be7189f8f45ccb329ef5d2fd8da8d
I20250625 13:52:01.996126 1747 leader_election.cc:290] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:01.996641 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6be7189f8f45ccb329ef5d2fd8da8d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:01.996898 1049 raft_consensus.cc:3058] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:01.997783 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
I20250625 13:52:01.998968 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_88d5d2614816414f8a45b3526e1608d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:01.999130 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 19017cc372af43b2b5074a8990375ede with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.000186 1049 raft_consensus.cc:2466] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
W20250625 13:52:02.000511 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88d5d2614816414f8a45b3526e1608d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.000808 849 leader_election.cc:304] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:02.001956 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu with pid 1222
I20250625 13:52:02.005121 1710 ts_tablet_manager.cc:1918] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.005280 1710 log.cc:1199] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/799f2edd960d45c1ace9195283a042e2
I20250625 13:52:02.005697 1710 ts_tablet_manager.cc:1939] T 799f2edd960d45c1ace9195283a042e2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:02.007373 1168 ts_tablet_manager.cc:1918] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:02.010427 1168 log.cc:1199] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/7108932fa8a241819ca90674a64b285e
I20250625 13:52:02.011065 1165 ts_tablet_manager.cc:1918] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.011442 1165 log.cc:1199] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/643928ca4fc94ce58431a0525f2abb99
I20250625 13:52:02.011829 1165 ts_tablet_manager.cc:1939] T 643928ca4fc94ce58431a0525f2abb99 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.012147 1168 ts_tablet_manager.cc:1939] T 7108932fa8a241819ca90674a64b285e P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:02.012755 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 493) took 1234 ms. Trace:
I20250625 13:52:02.012920 1165 rpcz_store.cc:276] 0625 13:52:00.778363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.778412 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:01.798783 (+1020371us) ts_tablet_manager.cc:1145] Deleting tablet 643928ca4fc94ce58431a0525f2abb99
0625 13:52:01.798796 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.893990 (+ 95194us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.010922 (+116932us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.012699 (+ 1777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020168}
I20250625 13:52:02.013075 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 799f2edd960d45c1ace9195283a042e2 (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]) successfully deleted
I20250625 13:52:02.013428 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 643928ca4fc94ce58431a0525f2abb99 (table default.Table_ad4e07371e3b42c5b05954f9537fd6b7 [id=c861ee66493e47ec97ad3aa9a6495499]) successfully deleted
I20250625 13:52:02.013823 1165 tablet_replica.cc:331] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:02.014029 1165 raft_consensus.cc:2241] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.014218 1165 raft_consensus.cc:2270] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.014760 1165 ts_tablet_manager.cc:1905] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.015372 1710 tablet_replica.cc:331] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:02.015343 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 17b172c7631048ceb97ccbea896f7f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17b172c7631048ceb97ccbea896f7f5b already in progress: creating tablet
W20250625 13:52:02.015771 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a29708fb688a4a65b84ae2564a040b4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a29708fb688a4a65b84ae2564a040b4c already in progress: creating tablet
I20250625 13:52:02.016517 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 253) took 5709 ms. Trace:
I20250625 13:52:02.016629 1168 rpcz_store.cc:276] 0625 13:51:56.306481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.307478 (+ 997us) service_pool.cc:225] Handling call
0625 13:52:01.963468 (+5655990us) ts_tablet_manager.cc:1145] Deleting tablet 7108932fa8a241819ca90674a64b285e
0625 13:52:01.963477 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:01.989695 (+ 26218us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.007226 (+ 17531us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.016419 (+ 9193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5621442}
I20250625 13:52:02.017009 1168 tablet_replica.cc:331] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:02.017423 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 7108932fa8a241819ca90674a64b285e (table default.Table_194007e5294849f9a6734e39b20d4fff [id=41a6138bc94a4a2790635b4ef0757c78]) successfully deleted
I20250625 13:52:02.017592 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f (DEFAULT_TABLE table=default.Table_5c8024face7142ac823c087d60f8b7b2 [id=b7d5479b4dd24985ba0f709a9ff28dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.018170 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.018754 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f (DEFAULT_TABLE table=default.Table_5c8024face7142ac823c087d60f8b7b2 [id=b7d5479b4dd24985ba0f709a9ff28dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.019140 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.023113 1747 raft_consensus.cc:2802] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:02.023327 1747 raft_consensus.cc:695] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:02.023835 1747 consensus_queue.cc:237] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.025328 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f7ba037477124796ab2a66a351968c19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.025898 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7ba037477124796ab2a66a351968c19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.026211 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 1a525544e77b4a96a79c8876c48bfae1 (DEFAULT_TABLE table=default.Table_07eaa93fdfa1484f98c567581341f0e9 [id=6980d122c3da49cdac9225d059385366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.026608 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a525544e77b4a96a79c8876c48bfae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.030615 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 8ebe47e363ce4a6c8129169bc6777e63 (DEFAULT_TABLE table=default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98 [id=e06a5b87bfb44b23beaeee29abf42788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.031004 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ebe47e363ce4a6c8129169bc6777e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.032455 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 48645c913fc24c03a922cf6ea7531022
I20250625 13:52:02.033149 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d1099e25efdb46c084ac0792c48cc54a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:02.033882 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 (DEFAULT_TABLE table=default.Table_fc57b2c273df4835bd06effe454ccc27 [id=a9e36e91e1ae4695a68daad3d1cada49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.034268 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.034340 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1099e25efdb46c084ac0792c48cc54a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.038552 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 71871e2231d64b92b73a4840c1222655 (DEFAULT_TABLE table=default.Table_c7c83caf0473417da3a8fc96412d92f0 [id=23aa99864007411ea7a81a5d5bdad4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.038971 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71871e2231d64b92b73a4840c1222655. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.039798 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 553af7c7326240838f37e3861d4e3357
I20250625 13:52:02.040287 588 catalog_manager.cc:6652] ReplaceTablet: tablet 553af7c7326240838f37e3861d4e3357 deleted and replaced by tablet bf7463962f39468c85fb3dc2edcf608f
I20250625 13:52:02.042088 1710 raft_consensus.cc:2241] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.042290 1710 raft_consensus.cc:2270] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.017208 1168 raft_consensus.cc:2241] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.042645 1168 raft_consensus.cc:2270] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.043188 1168 ts_tablet_manager.cc:1905] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.043466 1171 tablet_bootstrap.cc:492] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:02.043612 1171 ts_tablet_manager.cc:1397] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.176s user 0.007s sys 0.009s
I20250625 13:52:02.044462 1171 raft_consensus.cc:357] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.044667 1171 raft_consensus.cc:383] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:02.044775 1171 raft_consensus.cc:738] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:02.046470 1171 consensus_queue.cc:260] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.047101 1171 ts_tablet_manager.cc:1428] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:02.047272 888 tablet_service.cc:1468] Processing CreateTablet for tablet 71871e2231d64b92b73a4840c1222655 (DEFAULT_TABLE table=default.Table_c7c83caf0473417da3a8fc96412d92f0 [id=23aa99864007411ea7a81a5d5bdad4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.047438 1171 tablet_bootstrap.cc:492] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:02.047659 879 tablet_service.cc:1468] Processing CreateTablet for tablet 1a525544e77b4a96a79c8876c48bfae1 (DEFAULT_TABLE table=default.Table_07eaa93fdfa1484f98c567581341f0e9 [id=6980d122c3da49cdac9225d059385366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.047735 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71871e2231d64b92b73a4840c1222655. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.048045 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a525544e77b4a96a79c8876c48bfae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.051666 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8ebe47e363ce4a6c8129169bc6777e63 (DEFAULT_TABLE table=default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98 [id=e06a5b87bfb44b23beaeee29abf42788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.052008 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ebe47e363ce4a6c8129169bc6777e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.053584 1171 tablet_bootstrap.cc:654] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:02.055646 884 tablet_service.cc:1468] Processing CreateTablet for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 (DEFAULT_TABLE table=default.Table_fc57b2c273df4835bd06effe454ccc27 [id=a9e36e91e1ae4695a68daad3d1cada49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.056018 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.057929 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5df71374672448ce89f07be4d9f97e41" } new_table_name: "default.Table_e6dee677a6314593b3557d7e48d1504f" modify_external_catalogs: true
I20250625 13:52:02.059298 892 tablet_service.cc:1468] Processing CreateTablet for tablet 422f4d02482942dba453ff560890137b (DEFAULT_TABLE table=default.Table_a091a655459e4488b9f5d5798673fd42 [id=6f7e0665d085467b8c15379b7596a5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.060515 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422f4d02482942dba453ff560890137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.017593 755 tablet_service.cc:1468] Processing CreateTablet for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f (DEFAULT_TABLE table=default.Table_5c8024face7142ac823c087d60f8b7b2 [id=b7d5479b4dd24985ba0f709a9ff28dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.021471 754 tablet_service.cc:1468] Processing CreateTablet for tablet 71871e2231d64b92b73a4840c1222655 (DEFAULT_TABLE table=default.Table_c7c83caf0473417da3a8fc96412d92f0 [id=23aa99864007411ea7a81a5d5bdad4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.021803 753 tablet_service.cc:1468] Processing CreateTablet for tablet 8ebe47e363ce4a6c8129169bc6777e63 (DEFAULT_TABLE table=default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98 [id=e06a5b87bfb44b23beaeee29abf42788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.022159 744 tablet_service.cc:1468] Processing CreateTablet for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 (DEFAULT_TABLE table=default.Table_fc57b2c273df4835bd06effe454ccc27 [id=a9e36e91e1ae4695a68daad3d1cada49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.061378 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.061555 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.062952 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_180b708af9884741956d0cb3898f9efd" } new_table_name: "default.Table_2dbd2e7b0a624bcb99f8edc7918b558d" modify_external_catalogs: true
I20250625 13:52:02.059439 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 48645c913fc24c03a922cf6ea7531022 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.059512 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 553af7c7326240838f37e3861d4e3357 with delete_type TABLET_DATA_DELETED (replaced by tablet bf7463962f39468c85fb3dc2edcf608f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.066357 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 422f4d02482942dba453ff560890137b (DEFAULT_TABLE table=default.Table_a091a655459e4488b9f5d5798673fd42 [id=6f7e0665d085467b8c15379b7596a5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.066910 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422f4d02482942dba453ff560890137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.022490 746 tablet_service.cc:1468] Processing CreateTablet for tablet 422f4d02482942dba453ff560890137b (DEFAULT_TABLE table=default.Table_a091a655459e4488b9f5d5798673fd42 [id=6f7e0665d085467b8c15379b7596a5f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.067691 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422f4d02482942dba453ff560890137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.071115 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57
I20250625 13:52:02.071612 597 catalog_manager.cc:6652] ReplaceTablet: tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 deleted and replaced by tablet e29bade90b184d2483cb1fe85033a03c
I20250625 13:52:02.023222 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8cc353b88a124fac801b8898b5a5f72c (DEFAULT_TABLE table=default.Table_914aeef494b1417b96fd83b69c631c69 [id=39022c6f5aec4ac89c3df7c8b316e1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.072069 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc353b88a124fac801b8898b5a5f72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.073235 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 with delete_type TABLET_DATA_DELETED (replaced by tablet e29bade90b184d2483cb1fe85033a03c) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.073953 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 8cc353b88a124fac801b8898b5a5f72c (DEFAULT_TABLE table=default.Table_914aeef494b1417b96fd83b69c631c69 [id=39022c6f5aec4ac89c3df7c8b316e1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.074563 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc353b88a124fac801b8898b5a5f72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.075075 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 with delete_type TABLET_DATA_DELETED (replaced by tablet e29bade90b184d2483cb1fe85033a03c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.023643 743 tablet_service.cc:1468] Processing CreateTablet for tablet 1a525544e77b4a96a79c8876c48bfae1 (DEFAULT_TABLE table=default.Table_07eaa93fdfa1484f98c567581341f0e9 [id=6980d122c3da49cdac9225d059385366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.033710 1166 tablet_bootstrap.cc:492] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:02.034039 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 48645c913fc24c03a922cf6ea7531022 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.041585 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 553af7c7326240838f37e3861d4e3357 with delete_type TABLET_DATA_DELETED (replaced by tablet bf7463962f39468c85fb3dc2edcf608f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.076836 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71871e2231d64b92b73a4840c1222655. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.076843 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7e1bff138de841eda69caac9be5438bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.077436 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e1bff138de841eda69caac9be5438bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.078306 1166 ts_tablet_manager.cc:1397] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.132s user 0.002s sys 0.002s
I20250625 13:52:02.079135 1166 raft_consensus.cc:357] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.079366 1166 raft_consensus.cc:383] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:02.079444 1166 raft_consensus.cc:738] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:02.079690 1166 consensus_queue.cc:260] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.080229 1166 ts_tablet_manager.cc:1428] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:02.080588 1166 tablet_bootstrap.cc:492] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:02.081821 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ebe47e363ce4a6c8129169bc6777e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.083142 894 tablet_service.cc:1468] Processing CreateTablet for tablet 8cc353b88a124fac801b8898b5a5f72c (DEFAULT_TABLE table=default.Table_914aeef494b1417b96fd83b69c631c69 [id=39022c6f5aec4ac89c3df7c8b316e1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.083643 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc353b88a124fac801b8898b5a5f72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.083894 1748 raft_consensus.cc:491] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.084038 1748 raft_consensus.cc:513] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.085330 1748 leader_election.cc:290] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:02.085992 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9c1deb52366c404cbdc08076d4bcb3d1
I20250625 13:52:02.086153 1168 ts_tablet_manager.cc:1918] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.086328 1168 log.cc:1199] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/13620f6ca65342b4ad7f1ba4b437bde3
I20250625 13:52:02.086727 1168 ts_tablet_manager.cc:1939] T 13620f6ca65342b4ad7f1ba4b437bde3 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:02.086992 1710 ts_tablet_manager.cc:1905] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.087100 1518 raft_consensus.cc:491] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.087283 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.087325 1518 raft_consensus.cc:513] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.088214 1518 leader_election.cc:290] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:02.089594 983 leader_election.cc:343] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 85c72f9ae92a4537979fb479eb4467c4
I20250625 13:52:02.090543 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 254) took 5783 ms. Trace:
I20250625 13:52:02.090746 1168 rpcz_store.cc:276] 0625 13:51:56.307004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.310846 (+ 3842us) service_pool.cc:225] Handling call
0625 13:52:02.016918 (+5706072us) ts_tablet_manager.cc:1145] Deleting tablet 13620f6ca65342b4ad7f1ba4b437bde3
0625 13:52:02.016929 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.055196 (+ 38267us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.086011 (+ 30815us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.090497 (+ 4486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5671486}
I20250625 13:52:02.091279 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 13620f6ca65342b4ad7f1ba4b437bde3 (table default.Table_6199817767f74031b8a500d21174d5a6 [id=bbf70badd2ea45ae88881afe31a3de40]) successfully deleted
I20250625 13:52:02.091668 1168 tablet_replica.cc:331] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:02.093277 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a525544e77b4a96a79c8876c48bfae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.094691 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0231e5cbd04647a9911175976259ce4b" } new_table_name: "default.Table_b32fea3ffed64728b8b1b4a86b44c628" modify_external_catalogs: true
I20250625 13:52:02.097325 884 tablet_service.cc:1468] Processing CreateTablet for tablet cae3268035724c5396c59125f5c6d6fd (DEFAULT_TABLE table=default.Table_550cc88382f845bcad839c4bcd436274 [id=1022cdcee2c64f388a8abc8f6e48ce7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.097905 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae3268035724c5396c59125f5c6d6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.101653 1669 raft_consensus.cc:491] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.101831 1669 raft_consensus.cc:513] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:02.102855 581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.62:33015 (error 108)
I20250625 13:52:02.103710 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "991bb428def545fd979e9f1a80ce4d18" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:02.111469 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6515cc13bb054b8d89752a85d63f6996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.112039 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6515cc13bb054b8d89752a85d63f6996 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.112340 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.112655 758 tablet_service.cc:1468] Processing CreateTablet for tablet cae3268035724c5396c59125f5c6d6fd (DEFAULT_TABLE table=default.Table_550cc88382f845bcad839c4bcd436274 [id=1022cdcee2c64f388a8abc8f6e48ce7a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:02.113335 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 669de0b7c93a4efea8623dab2e462c8d consensus state has no leader
I20250625 13:52:02.113590 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 4c9efc7631df4ad29112a41f229ae2e9 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:02.113806 1669 leader_election.cc:290] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.115074 878 tablet_service.cc:1468] Processing CreateTablet for tablet 22ee97e6804a4b9fb564ed6f226bc1e0 (DEFAULT_TABLE table=default.Table_70ec24bdd5764fa1893a92a1e341a2ea [id=00360a43d057493bab6d55d317094943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.115485 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22ee97e6804a4b9fb564ed6f226bc1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.116101 752 tablet_service.cc:1468] Processing CreateTablet for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 (DEFAULT_TABLE table=default.Table_1f9d1b5465174503915b895238d69e85 [id=45adfcbffc314c94bb7ad5b588c7728c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:02.118381 985 leader_election.cc:343] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 85c72f9ae92a4537979fb479eb4467c4
I20250625 13:52:02.118547 985 leader_election.cc:304] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:02.119035 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 with delete_type TABLET_DATA_DELETED (replaced by tablet e29bade90b184d2483cb1fe85033a03c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.119743 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "991bb428def545fd979e9f1a80ce4d18" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.120044 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 553af7c7326240838f37e3861d4e3357 with delete_type TABLET_DATA_DELETED (replaced by tablet bf7463962f39468c85fb3dc2edcf608f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.120273 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93f449c80b604a6498f6fd1d1ba4770a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.111897 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 48645c913fc24c03a922cf6ea7531022 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.122732 1049 raft_consensus.cc:2466] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:02.122900 851 leader_election.cc:343] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 93f449c80b604a6498f6fd1d1ba4770a
I20250625 13:52:02.124881 1018 tablet_service.cc:1468] Processing CreateTablet for tablet cae3268035724c5396c59125f5c6d6fd (DEFAULT_TABLE table=default.Table_550cc88382f845bcad839c4bcd436274 [id=1022cdcee2c64f388a8abc8f6e48ce7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.125329 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae3268035724c5396c59125f5c6d6fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.126606 1759 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 13:52:02.122868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:33015 (local address 127.0.0.1:36258)
0625 13:52:02.126108 (+ 3240us) negotiation.cc:107] Waiting for socket to connect
0625 13:52:02.126279 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":3111,"threads_started":1}
I20250625 13:52:02.126662 1518 raft_consensus.cc:2747] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.127517 878 tablet_service.cc:1468] Processing CreateTablet for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 (DEFAULT_TABLE table=default.Table_1f9d1b5465174503915b895238d69e85 [id=45adfcbffc314c94bb7ad5b588c7728c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.127995 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0346ed4f8d3f46aa84cbabbcadcebf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.127996 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 (DEFAULT_TABLE table=default.Table_1f9d1b5465174503915b895238d69e85 [id=45adfcbffc314c94bb7ad5b588c7728c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.128363 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cae3268035724c5396c59125f5c6d6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.128770 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0346ed4f8d3f46aa84cbabbcadcebf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.131450 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0346ed4f8d3f46aa84cbabbcadcebf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.130335 1669 raft_consensus.cc:491] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.132995 1669 raft_consensus.cc:513] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.134696 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.135548 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 with delete_type TABLET_DATA_DELETED (replaced by tablet 86af0ea6648045a1ac4ba903d20849b7) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.136034 886 tablet_service.cc:1468] Processing CreateTablet for tablet 1f47260bf0ad41fdbfbad50380a12547 (DEFAULT_TABLE table=default.Table_56560af1ef3144d4a8daed436f4b0505 [id=851a1fc392994dbab495a3a187dfa074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.136437 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f47260bf0ad41fdbfbad50380a12547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.136610 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3376217fa08d41a1a3f44ca53dc1d43f (DEFAULT_TABLE table=default.Table_35471795fa7d4688ae966d55ca175371 [id=47c01d70221c45bcabe0185a84119ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.136962 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1f47260bf0ad41fdbfbad50380a12547 (DEFAULT_TABLE table=default.Table_56560af1ef3144d4a8daed436f4b0505 [id=851a1fc392994dbab495a3a187dfa074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.137560 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 22ee97e6804a4b9fb564ed6f226bc1e0 (DEFAULT_TABLE table=default.Table_70ec24bdd5764fa1893a92a1e341a2ea [id=00360a43d057493bab6d55d317094943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.137653 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f47260bf0ad41fdbfbad50380a12547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.138044 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 1f47260bf0ad41fdbfbad50380a12547 (DEFAULT_TABLE table=default.Table_56560af1ef3144d4a8daed436f4b0505 [id=851a1fc392994dbab495a3a187dfa074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.138850 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f47260bf0ad41fdbfbad50380a12547. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.138887 755 tablet_service.cc:1468] Processing CreateTablet for tablet 22ee97e6804a4b9fb564ed6f226bc1e0 (DEFAULT_TABLE table=default.Table_70ec24bdd5764fa1893a92a1e341a2ea [id=00360a43d057493bab6d55d317094943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.139220 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22ee97e6804a4b9fb564ed6f226bc1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.140241 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22ee97e6804a4b9fb564ed6f226bc1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.141883 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 3376217fa08d41a1a3f44ca53dc1d43f (DEFAULT_TABLE table=default.Table_35471795fa7d4688ae966d55ca175371 [id=47c01d70221c45bcabe0185a84119ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.142196 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3376217fa08d41a1a3f44ca53dc1d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.142567 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3376217fa08d41a1a3f44ca53dc1d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.144244 888 tablet_service.cc:1468] Processing CreateTablet for tablet 3376217fa08d41a1a3f44ca53dc1d43f (DEFAULT_TABLE table=default.Table_35471795fa7d4688ae966d55ca175371 [id=47c01d70221c45bcabe0185a84119ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.132740 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_796954f1442545d2aa7d7ffaf74e29d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:02.144650 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3376217fa08d41a1a3f44ca53dc1d43f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.144987 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_796954f1442545d2aa7d7ffaf74e29d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:02.145891 1756 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 13:52:02.089111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:33015 (local address 127.0.104.2:56493)
0625 13:52:02.145336 (+ 56225us) negotiation.cc:107] Waiting for socket to connect
0625 13:52:02.145526 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":56080,"threads_started":1}
I20250625 13:52:02.141817 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet fb4973cbffc743dba98d7b7c9f76be3f
W20250625 13:52:02.146481 849 consensus_peers.cc:487] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d524089e6ab044ec961cfc9f587e8f86. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:02.146685 849 leader_election.cc:304] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:02.147037 1747 raft_consensus.cc:2802] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:02.147127 1747 raft_consensus.cc:491] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:02.147230 1747 raft_consensus.cc:3058] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:02.148275 757 tablet_service.cc:1468] Processing CreateTablet for tablet 3f06030f45284c5a9929a4c4513004ba (DEFAULT_TABLE table=default.Table_938b272613694e4bb9b163d19e37053a [id=fe62358f99bf4b89a66fdfdcd4670c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.148635 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f06030f45284c5a9929a4c4513004ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.149945 1747 raft_consensus.cc:513] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.150840 1747 leader_election.cc:290] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:02.152001 581 consensus_peers.cc:487] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 -> Peer 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015): Couldn't send request to peer 3fec9d8a79054a5991ae90d8a3101e32. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:02.152359 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93f449c80b604a6498f6fd1d1ba4770a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:02.152962 750 tablet_service.cc:1468] Processing CreateTablet for tablet c9a82adc5aa045f49e092dababafe4f8 (DEFAULT_TABLE table=default.Table_ecb211456c1e44e08827d9509ba961d2 [id=82ac4f198e8242fc9a95787dc3331c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.153064 1049 raft_consensus.cc:3058] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:02.154393 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a82adc5aa045f49e092dababafe4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.154949 1669 leader_election.cc:290] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:02.156802 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet fb4973cbffc743dba98d7b7c9f76be3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.157024 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c9a82adc5aa045f49e092dababafe4f8 (DEFAULT_TABLE table=default.Table_ecb211456c1e44e08827d9509ba961d2 [id=82ac4f198e8242fc9a95787dc3331c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.159973 741 tablet_service.cc:1468] Processing CreateTablet for tablet fd7a6587aa674a0eaaa107e2a2f673a3 (DEFAULT_TABLE table=default.Table_7d1b445e65344381aac8680dad76a4b2 [id=f63d1eacefa9422a97784e55c027ac35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.160409 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7a6587aa674a0eaaa107e2a2f673a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.167771 747 tablet_service.cc:1468] Processing CreateTablet for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 (DEFAULT_TABLE table=default.Table_f04ee0e6b3324e169e99b807c1faa3a0 [id=456ba28e3d4347778ffce07f1aaf2d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.168185 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e147e09b5244a8b8f52e8d10c3db4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.168552 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a82adc5aa045f49e092dababafe4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.171604 745 tablet_service.cc:1468] Processing CreateTablet for tablet d5abd4d83514458c95ec62290b709bc3 (DEFAULT_TABLE table=default.Table_ba34195043ab4dd79dea215f57cbb3cc [id=f818da66b13b41109244d6d0139ccd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.171954 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5abd4d83514458c95ec62290b709bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.176196 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ecb211456c1e44e08827d9509ba961d2" } new_table_name: "default.Table_d8054fa43c994aab98b1d5541d0be6f9" modify_external_catalogs: true
I20250625 13:52:02.178303 740 tablet_service.cc:1468] Processing CreateTablet for tablet 6fcc17f839504d6b85526b638763ec5b (DEFAULT_TABLE table=default.Table_c9c75aecd94048888373fbb8cc3efdbd [id=897e6c8a77b7488eb59a361168649484]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.178673 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcc17f839504d6b85526b638763ec5b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.181833 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af704cdf44ef42f7b5ea10b199d0f017 consensus state has no leader
W20250625 13:52:02.182240 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8b30c229c3445b683a5015f069eca1c consensus state has no leader
I20250625 13:52:02.182123 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3f06030f45284c5a9929a4c4513004ba (DEFAULT_TABLE table=default.Table_938b272613694e4bb9b163d19e37053a [id=fe62358f99bf4b89a66fdfdcd4670c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.182621 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f06030f45284c5a9929a4c4513004ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.183703 754 tablet_service.cc:1515] Processing DeleteTablet for tablet fb4973cbffc743dba98d7b7c9f76be3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.184983 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1480ea8e34de4a38b7f925b0079ac328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.185525 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1480ea8e34de4a38b7f925b0079ac328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.186813 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1ffaf5826eda45159d7cef2f2505383a" } modify_external_catalogs: true
I20250625 13:52:02.186995 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1ffaf5826eda45159d7cef2f2505383a" } modify_external_catalogs: true
I20250625 13:52:02.186822 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_531f237d57114b2483ae3cf8d4b94659" } modify_external_catalogs: true
I20250625 13:52:02.188019 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_531f237d57114b2483ae3cf8d4b94659" } modify_external_catalogs: true
I20250625 13:52:02.192569 1015 tablet_service.cc:1468] Processing CreateTablet for tablet fd7a6587aa674a0eaaa107e2a2f673a3 (DEFAULT_TABLE table=default.Table_7d1b445e65344381aac8680dad76a4b2 [id=f63d1eacefa9422a97784e55c027ac35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.193709 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7a6587aa674a0eaaa107e2a2f673a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.196475 1049 raft_consensus.cc:2466] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:02.197427 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 (DEFAULT_TABLE table=default.Table_f04ee0e6b3324e169e99b807c1faa3a0 [id=456ba28e3d4347778ffce07f1aaf2d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.190439 756 tablet_service.cc:1468] Processing CreateTablet for tablet 7db450dc63bd4b6caf074c67490cd175 (DEFAULT_TABLE table=default.Table_6a3e9fc3400f4884a44a9054a64afe1c [id=538d4f6a5f6347a38d1aa24abdba5d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.199347 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db450dc63bd4b6caf074c67490cd175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.198175 751 tablet_service.cc:1468] Processing CreateTablet for tablet a89f9ffb5e4643f8915e9fd69dd3e842 (DEFAULT_TABLE table=default.Table_e11bb2795f394504b85419cbea5cb249 [id=862bc721f3cf4ba794a56b6ea8ce53a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.203228 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89f9ffb5e4643f8915e9fd69dd3e842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.203363 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d5abd4d83514458c95ec62290b709bc3 (DEFAULT_TABLE table=default.Table_ba34195043ab4dd79dea215f57cbb3cc [id=f818da66b13b41109244d6d0139ccd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.203924 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 6fcc17f839504d6b85526b638763ec5b (DEFAULT_TABLE table=default.Table_c9c75aecd94048888373fbb8cc3efdbd [id=897e6c8a77b7488eb59a361168649484]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.204331 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e147e09b5244a8b8f52e8d10c3db4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.205695 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcc17f839504d6b85526b638763ec5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.206988 1669 raft_consensus.cc:491] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.207116 1669 raft_consensus.cc:513] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.207417 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 7db450dc63bd4b6caf074c67490cd175 (DEFAULT_TABLE table=default.Table_6a3e9fc3400f4884a44a9054a64afe1c [id=538d4f6a5f6347a38d1aa24abdba5d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.207784 1669 leader_election.cc:290] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.208460 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db450dc63bd4b6caf074c67490cd175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.210773 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5abd4d83514458c95ec62290b709bc3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.211862 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15c81fab8e0b442e8d39556c1bce742e consensus state has no leader
I20250625 13:52:02.214779 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ce91dd339b4c4261bdc2fe9e91b0d413"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.215418 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce91dd339b4c4261bdc2fe9e91b0d413 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.216030 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a53e9e99051944a1bc66244b53112731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.216483 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a53e9e99051944a1bc66244b53112731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.220155 754 tablet_service.cc:1468] Processing CreateTablet for tablet a85fa343e30e4e838ab81cd775c298ba (DEFAULT_TABLE table=default.Table_688372ad7a204de3aa17a0fcf7c0a34f [id=b3fbf1334d9a4233ac9adf46374ff1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.220680 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85fa343e30e4e838ab81cd775c298ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.227147 851 leader_election.cc:343] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 93f449c80b604a6498f6fd1d1ba4770a
W20250625 13:52:02.227420 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 71249.4us
I20250625 13:52:02.233404 749 tablet_service.cc:1468] Processing CreateTablet for tablet 39540b81fd4746858e65be49ded0e1df (DEFAULT_TABLE table=default.Table_72f464f3d7e7440da8df36ec84428a8d [id=5c2bbcb00fd34502a1771eb0c274865c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.233840 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39540b81fd4746858e65be49ded0e1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.236164 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a89f9ffb5e4643f8915e9fd69dd3e842 (DEFAULT_TABLE table=default.Table_e11bb2795f394504b85419cbea5cb249 [id=862bc721f3cf4ba794a56b6ea8ce53a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.236557 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89f9ffb5e4643f8915e9fd69dd3e842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.241524 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1ffafeb50d654290963bf41dfeb57f55" } modify_external_catalogs: true
I20250625 13:52:02.241807 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1ffafeb50d654290963bf41dfeb57f55" } modify_external_catalogs: true
I20250625 13:52:02.245700 753 tablet_service.cc:1468] Processing CreateTablet for tablet 412fabb4a60647e7992756879596e76b (DEFAULT_TABLE table=default.Table_44d60eefccee45558969e859fa7897c6 [id=a7684258f7194406a581a11e16803e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.246294 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412fabb4a60647e7992756879596e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.255023 1518 raft_consensus.cc:491] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.255236 1518 raft_consensus.cc:513] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.256690 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50786a4c25274ffb8f9910c5dafe3933" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:52:02.258457 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9a82adc5aa045f49e092dababafe4f8 consensus state has no leader
I20250625 13:52:02.262298 743 tablet_service.cc:1468] Processing CreateTablet for tablet a6ff3b675aba42adb035ff5d1404a889 (DEFAULT_TABLE table=default.Table_472e8cfcd53c4be2b738c172ae9fb892 [id=71cae1b0a11c4a8584eb9c72586ba538]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.262755 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ff3b675aba42adb035ff5d1404a889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.266547 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 412fabb4a60647e7992756879596e76b (DEFAULT_TABLE table=default.Table_44d60eefccee45558969e859fa7897c6 [id=a7684258f7194406a581a11e16803e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.266811 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 39540b81fd4746858e65be49ded0e1df (DEFAULT_TABLE table=default.Table_72f464f3d7e7440da8df36ec84428a8d [id=5c2bbcb00fd34502a1771eb0c274865c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.277825 1518 leader_election.cc:290] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.280383 749 tablet_service.cc:1468] Processing CreateTablet for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 (DEFAULT_TABLE table=default.Table_8cdfd947725147e294c585373c3c4fc7 [id=6177dd08dae34295b2aac9711e530216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.280948 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eabd67b9ceb74bd1a5f4f115c7db0e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.283679 750 tablet_service.cc:1468] Processing CreateTablet for tablet 755a542d04dd44a4948c6657b65ce443 (DEFAULT_TABLE table=default.Table_7e3cd23ead3f4afa889078106d9b4711 [id=3bfbc121faf44d6691a73639838194c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.284263 756 tablet_service.cc:1468] Processing CreateTablet for tablet f3ce31199dff4300a63930b1c8e6dc4e (DEFAULT_TABLE table=default.Table_20830d90ca194d2d8a472b90228ae5b0 [id=2b83edcabfbb479d936e8802a569beb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.288945 758 tablet_service.cc:1468] Processing CreateTablet for tablet fd4142572b344456a0d8248be914ad8f (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.292870 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5c8024face7142ac823c087d60f8b7b2" } modify_external_catalogs: true
I20250625 13:52:02.293079 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5c8024face7142ac823c087d60f8b7b2" } modify_external_catalogs: true
I20250625 13:52:02.284368 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755a542d04dd44a4948c6657b65ce443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.296044 849 leader_election.cc:304] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:02.297941 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ce31199dff4300a63930b1c8e6dc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.297940 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6e0f2fa5d5614ad394e6e12ec62814d4
W20250625 13:52:02.299155 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 85058.1us
I20250625 13:52:02.300050 1748 raft_consensus.cc:2802] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:02.301266 1748 raft_consensus.cc:695] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:02.301787 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4142572b344456a0d8248be914ad8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.304054 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50786a4c25274ffb8f9910c5dafe3933" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:02.304358 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.305688 1166 tablet_bootstrap.cc:654] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:02.306216 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3f06030f45284c5a9929a4c4513004ba (DEFAULT_TABLE table=default.Table_938b272613694e4bb9b163d19e37053a [id=fe62358f99bf4b89a66fdfdcd4670c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.306730 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f06030f45284c5a9929a4c4513004ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.306787 896 tablet_service.cc:1468] Processing CreateTablet for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 (DEFAULT_TABLE table=default.Table_f04ee0e6b3324e169e99b807c1faa3a0 [id=456ba28e3d4347778ffce07f1aaf2d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.307634 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 69672579a83c41d09ee7c78a6fa7d746
I20250625 13:52:02.301689 1748 consensus_queue.cc:237] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.309689 1168 raft_consensus.cc:2241] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:02.312618 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.319465 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0d633ac15e6e4b14b8fda5e3845c6c6d" } new_table_name: "default.Table_fd444f4e61ca4ecb9dc9ee9660e8e36b" modify_external_catalogs: true
I20250625 13:52:02.320195 878 tablet_service.cc:1468] Processing CreateTablet for tablet fd7a6587aa674a0eaaa107e2a2f673a3 (DEFAULT_TABLE table=default.Table_7d1b445e65344381aac8680dad76a4b2 [id=f63d1eacefa9422a97784e55c027ac35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.320360 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b507798c190f493d9c549c62c02bc868"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.320936 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b507798c190f493d9c549c62c02bc868 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.323079 882 tablet_service.cc:1468] Processing CreateTablet for tablet d5abd4d83514458c95ec62290b709bc3 (DEFAULT_TABLE table=default.Table_ba34195043ab4dd79dea215f57cbb3cc [id=f818da66b13b41109244d6d0139ccd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.323964 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e147e09b5244a8b8f52e8d10c3db4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.324298 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6fcc17f839504d6b85526b638763ec5b (DEFAULT_TABLE table=default.Table_c9c75aecd94048888373fbb8cc3efdbd [id=897e6c8a77b7488eb59a361168649484]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.324362 1747 raft_consensus.cc:491] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.332839 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_850e088a094149f5a9b02a69a960624b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.333674 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_850e088a094149f5a9b02a69a960624b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.342964 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8063122ea7b24b6790f5c3cc884e09be
I20250625 13:52:02.345934 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8063122ea7b24b6790f5c3cc884e09be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.349018 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_472e8cfcd53c4be2b738c172ae9fb892" } modify_external_catalogs: true
I20250625 13:52:02.349244 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_472e8cfcd53c4be2b738c172ae9fb892" } modify_external_catalogs: true
I20250625 13:52:02.351990 1166 tablet_bootstrap.cc:492] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:02.352288 1166 ts_tablet_manager.cc:1397] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.272s user 0.009s sys 0.001s
I20250625 13:52:02.353776 1166 raft_consensus.cc:357] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.354260 1166 raft_consensus.cc:383] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:02.354485 1166 raft_consensus.cc:738] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:02.354950 1166 consensus_queue.cc:260] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.355163 1669 raft_consensus.cc:491] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.355681 1669 raft_consensus.cc:513] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.356245 1166 ts_tablet_manager.cc:1428] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:02.356457 1669 leader_election.cc:290] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.357264 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5d4fa8abea8d4eee8651be9bf4b2b34f
I20250625 13:52:02.359110 1166 tablet_bootstrap.cc:492] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:02.362519 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.329501 893 tablet_service.cc:1468] Processing CreateTablet for tablet 412fabb4a60647e7992756879596e76b (DEFAULT_TABLE table=default.Table_44d60eefccee45558969e859fa7897c6 [id=a7684258f7194406a581a11e16803e25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.329751 895 tablet_service.cc:1468] Processing CreateTablet for tablet a6ff3b675aba42adb035ff5d1404a889 (DEFAULT_TABLE table=default.Table_472e8cfcd53c4be2b738c172ae9fb892 [id=71cae1b0a11c4a8584eb9c72586ba538]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.333235 887 tablet_service.cc:1468] Processing CreateTablet for tablet 755a542d04dd44a4948c6657b65ce443 (DEFAULT_TABLE table=default.Table_7e3cd23ead3f4afa889078106d9b4711 [id=3bfbc121faf44d6691a73639838194c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.336201 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.366515 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:02.366770 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: LEADER
I20250625 13:52:02.368175 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5abd4d83514458c95ec62290b709bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.368444 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcc17f839504d6b85526b638763ec5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.368692 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f13b94122bf947aab60dbf0eb035cc63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.369239 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f13b94122bf947aab60dbf0eb035cc63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.371099 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e170ed64e7174c39a3e21ceac0e51021"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.371623 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e170ed64e7174c39a3e21ceac0e51021 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.373875 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a85fa343e30e4e838ab81cd775c298ba (DEFAULT_TABLE table=default.Table_688372ad7a204de3aa17a0fcf7c0a34f [id=b3fbf1334d9a4233ac9adf46374ff1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.375847 983 leader_election.cc:304] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:02.382017 1165 ts_tablet_manager.cc:1918] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.382220 1165 log.cc:1199] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/bb1c7a85da844d34be805843bc60d350
I20250625 13:52:02.383675 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e11bb2795f394504b85419cbea5cb249" } new_table_name: "default.Table_5426c9800c16492fa6da1e38ccdf5139" modify_external_catalogs: true
I20250625 13:52:02.384402 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1e0aeab3aac8463aad52e947a2aadb14" } new_table_name: "default.Table_034b191549844f928bcd64e6d87483dd" modify_external_catalogs: true
I20250625 13:52:02.385267 1518 raft_consensus.cc:491] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:02.388489 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 653267b468ef4958b98af0ed6a7bf4d3 consensus state has no leader
I20250625 13:52:02.397063 1166 tablet_bootstrap.cc:654] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:02.400462 1518 raft_consensus.cc:513] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.401436 1518 leader_election.cc:290] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.399029 1017 tablet_service.cc:1468] Processing CreateTablet for tablet a6ff3b675aba42adb035ff5d1404a889 (DEFAULT_TABLE table=default.Table_472e8cfcd53c4be2b738c172ae9fb892 [id=71cae1b0a11c4a8584eb9c72586ba538]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.410032 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b1982b3d458a4586a464994f13dd1481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.410775 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1982b3d458a4586a464994f13dd1481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.328253 894 tablet_service.cc:1468] Processing CreateTablet for tablet c9a82adc5aa045f49e092dababafe4f8 (DEFAULT_TABLE table=default.Table_ecb211456c1e44e08827d9509ba961d2 [id=82ac4f198e8242fc9a95787dc3331c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.413478 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1452b75693a442568546fb6e23ff895e" } new_table_name: "default.Table_349a67a1a0284f84ac46a166fb498b07" modify_external_catalogs: true
W20250625 13:52:02.411046 711 connection.cc:537] client connection to 127.0.104.62:33015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:02.411405 1165 ts_tablet_manager.cc:1939] T bb1c7a85da844d34be805843bc60d350 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.416062 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 492) took 1637 ms. Trace:
I20250625 13:52:02.416261 1165 rpcz_store.cc:276] 0625 13:52:00.778237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.778709 (+ 472us) service_pool.cc:225] Handling call
0625 13:52:02.013733 (+1235024us) ts_tablet_manager.cc:1145] Deleting tablet bb1c7a85da844d34be805843bc60d350
0625 13:52:02.013748 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.327987 (+314239us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.381884 (+ 53897us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.416023 (+ 34139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234777}
I20250625 13:52:02.416888 1165 tablet_replica.cc:331] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:02.417269 1165 raft_consensus.cc:2241] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.417706 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet bb1c7a85da844d34be805843bc60d350 (table default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]) successfully deleted
I20250625 13:52:02.419946 1710 ts_tablet_manager.cc:1918] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.421144 1165 raft_consensus.cc:2270] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.421777 1165 ts_tablet_manager.cc:1905] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.421933 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f3ce31199dff4300a63930b1c8e6dc4e (DEFAULT_TABLE table=default.Table_20830d90ca194d2d8a472b90228ae5b0 [id=2b83edcabfbb479d936e8802a569beb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.422442 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:02.424201 1170 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.007s sys 0.043s
I20250625 13:52:02.421759 1023 tablet_service.cc:1468] Processing CreateTablet for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 (DEFAULT_TABLE table=default.Table_8cdfd947725147e294c585373c3c4fc7 [id=6177dd08dae34295b2aac9711e530216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.428043 1518 raft_consensus.cc:2747] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.432478 892 tablet_service.cc:1515] Processing DeleteTablet for tablet fb4973cbffc743dba98d7b7c9f76be3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.433943 1166 tablet_bootstrap.cc:492] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:02.434130 1166 ts_tablet_manager.cc:1397] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.002s
I20250625 13:52:02.434973 1166 raft_consensus.cc:357] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.435174 1166 raft_consensus.cc:383] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:52:02.436373 711 leader_election.cc:343] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.436362 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8063122ea7b24b6790f5c3cc884e09be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.436318 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_82770759b43b46a0850155827b18c124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:02.436553 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:02.436862 711 leader_election.cc:343] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:02.436967 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82770759b43b46a0850155827b18c124 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.437028 711 leader_election.cc:304] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:02.437378 1757 raft_consensus.cc:2747] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.438688 891 tablet_service.cc:1468] Processing CreateTablet for tablet 39540b81fd4746858e65be49ded0e1df (DEFAULT_TABLE table=default.Table_72f464f3d7e7440da8df36ec84428a8d [id=5c2bbcb00fd34502a1771eb0c274865c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.438856 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7a6587aa674a0eaaa107e2a2f673a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.439404 1747 raft_consensus.cc:513] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.435257 1166 raft_consensus.cc:738] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:02.440625 1166 consensus_queue.cc:260] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.441184 1166 ts_tablet_manager.cc:1428] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 13:52:02.441949 1168 pending_rounds.cc:70] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:02.446475 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ebff453083b745618b11918f62df6a0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.447029 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebff453083b745618b11918f62df6a0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.449364 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a6ff3b675aba42adb035ff5d1404a889
I20250625 13:52:02.451450 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a6ff3b675aba42adb035ff5d1404a889 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.453215 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 with delete_type TABLET_DATA_DELETED (replaced by tablet 32143e2497ee4d1b8c2e7cbfc6160ef4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.456401 1166 tablet_bootstrap.cc:492] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:02.456650 889 tablet_service.cc:1468] Processing CreateTablet for tablet f3ce31199dff4300a63930b1c8e6dc4e (DEFAULT_TABLE table=default.Table_20830d90ca194d2d8a472b90228ae5b0 [id=2b83edcabfbb479d936e8802a569beb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.460932 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dfa5acc20d2d4b34ba0a3503952923f5" } new_table_name: "default.Table_221603e7750d443aad1f5583b695b1c0" modify_external_catalogs: true
I20250625 13:52:02.463234 1165 ts_tablet_manager.cc:1918] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.467499 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 8063122ea7b24b6790f5c3cc884e09be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.475703 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f37637a5cbf94eb986a689ea87854c41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:02.476226 1165 log.cc:1199] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/15e58fbec94e4ff9b7ac9830b576fe7f
W20250625 13:52:02.476375 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f37637a5cbf94eb986a689ea87854c41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.476794 1165 ts_tablet_manager.cc:1939] T 15e58fbec94e4ff9b7ac9830b576fe7f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.478353 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 513) took 1010 ms. Trace:
I20250625 13:52:02.478520 1165 rpcz_store.cc:276] 0625 13:52:01.467493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.467698 (+ 205us) service_pool.cc:225] Handling call
0625 13:52:02.416823 (+949125us) ts_tablet_manager.cc:1145] Deleting tablet 15e58fbec94e4ff9b7ac9830b576fe7f
0625 13:52:02.416832 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.434121 (+ 17289us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.463114 (+ 28993us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.478294 (+ 15180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":945923}
I20250625 13:52:02.479048 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 15e58fbec94e4ff9b7ac9830b576fe7f (table default.Table_72b53bc133ea42568572d075e5493c2e [id=df6a0a5985da4e6e82eb4b99149e24ba]) successfully deleted
I20250625 13:52:02.479475 1165 tablet_replica.cc:331] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:02.479714 1165 raft_consensus.cc:2241] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.479996 1165 raft_consensus.cc:2270] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.480974 885 tablet_service.cc:1468] Processing CreateTablet for tablet a85fa343e30e4e838ab81cd775c298ba (DEFAULT_TABLE table=default.Table_688372ad7a204de3aa17a0fcf7c0a34f [id=b3fbf1334d9a4233ac9adf46374ff1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.481340 1165 ts_tablet_manager.cc:1905] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.481359 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.481670 1604 raft_consensus.cc:491] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.481868 1604 raft_consensus.cc:513] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.482740 1604 leader_election.cc:290] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:02.483031 881 tablet_service.cc:1468] Processing CreateTablet for tablet fd4142572b344456a0d8248be914ad8f (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:02.483198 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 67441795e22b445a90c88d76aa519ea5 (table default.Table_5d5a62a1608c45929c8fa1c853a06a27 [id=41e7cd4d77014eb0b0f1c2336c32bf1a], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 67441795e22b445a90c88d76aa519ea5 consensus state has no leader
I20250625 13:52:02.486136 883 tablet_service.cc:1468] Processing CreateTablet for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 (DEFAULT_TABLE table=default.Table_8cdfd947725147e294c585373c3c4fc7 [id=6177dd08dae34295b2aac9711e530216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.486425 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7c3f0ca6fa5482db7f57143911c838f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:02.489967 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7db450dc63bd4b6caf074c67490cd175 (DEFAULT_TABLE table=default.Table_6a3e9fc3400f4884a44a9054a64afe1c [id=538d4f6a5f6347a38d1aa24abdba5d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.490772 1165 ts_tablet_manager.cc:1918] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.491003 1165 log.cc:1199] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/04479dfb34584f0891c604ef72bf93b5
I20250625 13:52:02.491468 1165 ts_tablet_manager.cc:1939] T 04479dfb34584f0891c604ef72bf93b5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.492234 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 514) took 1024 ms. Trace:
I20250625 13:52:02.492390 1165 rpcz_store.cc:276] 0625 13:52:01.467645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.467896 (+ 251us) service_pool.cc:225] Handling call
0625 13:52:02.479383 (+1011487us) ts_tablet_manager.cc:1145] Deleting tablet 04479dfb34584f0891c604ef72bf93b5
0625 13:52:02.479393 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.486581 (+ 7188us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.490624 (+ 4043us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.492203 (+ 1579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008349}
I20250625 13:52:02.493036 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 04479dfb34584f0891c604ef72bf93b5 (table default.Table_c1e172984ca643a9b7c88d09f4d5221a [id=d0ac91ff8e7f473b8b2f938243067eac]) successfully deleted
I20250625 13:52:02.493389 1165 tablet_replica.cc:331] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:02.493613 1165 raft_consensus.cc:2241] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.493804 1165 raft_consensus.cc:2270] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.494511 1165 ts_tablet_manager.cc:1905] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.472321 1166 tablet_bootstrap.cc:654] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:02.500806 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_72f464f3d7e7440da8df36ec84428a8d" } modify_external_catalogs: true
I20250625 13:52:02.501029 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_72f464f3d7e7440da8df36ec84428a8d" } modify_external_catalogs: true
I20250625 13:52:02.502298 1165 ts_tablet_manager.cc:1918] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.502439 1165 log.cc:1199] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/f033bfa4925b4d1c971789eb77efd92c
I20250625 13:52:02.502789 1165 ts_tablet_manager.cc:1939] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.503363 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 515) took 1026 ms. Trace:
I20250625 13:52:02.503473 1165 rpcz_store.cc:276] 0625 13:52:01.476967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.477145 (+ 178us) service_pool.cc:225] Handling call
0625 13:52:02.493331 (+1016186us) ts_tablet_manager.cc:1145] Deleting tablet f033bfa4925b4d1c971789eb77efd92c
0625 13:52:02.493340 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.498435 (+ 5095us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.502199 (+ 3764us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.503268 (+ 1069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015975}
I20250625 13:52:02.503960 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet f033bfa4925b4d1c971789eb77efd92c (table default.Table_df8e6bfa3a4741fca89b09188c645be9 [id=e8ccbd9e2b83408b87689d6d6f26eb47]) successfully deleted
I20250625 13:52:02.509680 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e7baa9cc87143b0833a6bc32a193408" } modify_external_catalogs: true
I20250625 13:52:02.509889 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e7baa9cc87143b0833a6bc32a193408" } modify_external_catalogs: true
W20250625 13:52:02.511929 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a89f9ffb5e4643f8915e9fd69dd3e842 consensus state has no leader
I20250625 13:52:02.513319 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c8d20a37f388454db75fc0c021da9321"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.513845 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8d20a37f388454db75fc0c021da9321 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.515579 1165 tablet_replica.cc:331] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:02.515906 1165 raft_consensus.cc:2241] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.516184 1165 raft_consensus.cc:2270] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.516793 1165 ts_tablet_manager.cc:1905] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:02.519246 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3c75deb64c74f0282efeed827659ec2 consensus state has no leader
I20250625 13:52:02.524430 888 tablet_service.cc:1468] Processing CreateTablet for tablet a89f9ffb5e4643f8915e9fd69dd3e842 (DEFAULT_TABLE table=default.Table_e11bb2795f394504b85419cbea5cb249 [id=862bc721f3cf4ba794a56b6ea8ce53a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.533061 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9fda2b05f42e471fb687a7c46532b3e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.534072 711 leader_election.cc:343] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:02.535286 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fda2b05f42e471fb687a7c46532b3e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.539559 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 755a542d04dd44a4948c6657b65ce443 (DEFAULT_TABLE table=default.Table_7e3cd23ead3f4afa889078106d9b4711 [id=3bfbc121faf44d6691a73639838194c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.549153 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f4ab201a21c4415fa8b6ac70e579492b" } modify_external_catalogs: true
I20250625 13:52:02.549361 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f4ab201a21c4415fa8b6ac70e579492b" } modify_external_catalogs: true
W20250625 13:52:02.550257 985 leader_election.cc:343] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f73f420a4d7a438eadde3d6af64f0a11
I20250625 13:52:02.551151 1165 ts_tablet_manager.cc:1918] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:02.551728 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d25e4114a184bc99705042593ab4ab9 consensus state has no leader
I20250625 13:52:02.552881 1518 raft_consensus.cc:491] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.553562 1165 log.cc:1199] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/991bb428def545fd979e9f1a80ce4d18
I20250625 13:52:02.554123 1165 ts_tablet_manager.cc:1939] T 991bb428def545fd979e9f1a80ce4d18 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.555087 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 516) took 1008 ms. Trace:
I20250625 13:52:02.560038 1165 rpcz_store.cc:276] 0625 13:52:01.546328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.546385 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:02.515461 (+969076us) ts_tablet_manager.cc:1145] Deleting tablet 991bb428def545fd979e9f1a80ce4d18
0625 13:52:02.515475 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.538390 (+ 22915us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.550998 (+ 12608us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.555044 (+ 4046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":968815}
I20250625 13:52:02.560820 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 991bb428def545fd979e9f1a80ce4d18 (table default.Table_a21873ae3bbd4a1aaf287e80fe14c12e [id=fd9a1350f16e4c54a613cf8f7026240a]) successfully deleted
W20250625 13:52:02.561856 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 063a793e0d7747668d2728fdc2160b5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 063a793e0d7747668d2728fdc2160b5e already in progress: creating tablet
I20250625 13:52:02.562966 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4dd664d07c7f4b22a44352da05e41e94" } modify_external_catalogs: true
I20250625 13:52:02.563143 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4dd664d07c7f4b22a44352da05e41e94" } modify_external_catalogs: true
I20250625 13:52:02.562584 1165 tablet_replica.cc:331] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:02.563977 1165 raft_consensus.cc:2241] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.564673 1165 raft_consensus.cc:2270] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.572475 1165 ts_tablet_manager.cc:1905] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.574193 1747 leader_election.cc:290] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:02.575388 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3016763b9e294b9da2861fe97e57e241 consensus state has no leader
I20250625 13:52:02.577359 1510 raft_consensus.cc:491] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.577032 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6771d8e180fe48569a0e016db0cc9b51" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:02.578351 711 leader_election.cc:343] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.579241 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.580343 1510 raft_consensus.cc:513] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.580739 1168 pending_rounds.cc:77] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520602212093952 op_type: NO_OP noop_request { }
I20250625 13:52:02.582352 884 tablet_service.cc:1515] Processing DeleteTablet for tablet a6ff3b675aba42adb035ff5d1404a889 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.584367 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412fabb4a60647e7992756879596e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.585855 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f033bfa4925b4d1c971789eb77efd92c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:02.587348 711 leader_election.cc:343] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.589998 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_850e088a094149f5a9b02a69a960624b" } new_table_name: "default.Table_809755ea1ee14781acbef7ee9ccf5548" modify_external_catalogs: true
I20250625 13:52:02.590344 1168 raft_consensus.cc:2887] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20250625 13:52:02.591150 851 leader_election.cc:343] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cd4f0857f52449e1b26fde09c6c7c9d4
W20250625 13:52:02.591378 1171 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.106s sys 0.291s
I20250625 13:52:02.593014 1165 ts_tablet_manager.cc:1918] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.596398 1165 log.cc:1199] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/055f88e118454e32baa578cae9aceb0c
I20250625 13:52:02.596942 1165 ts_tablet_manager.cc:1939] T 055f88e118454e32baa578cae9aceb0c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.597177 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:02.597594 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 145542us
I20250625 13:52:02.598747 1757 raft_consensus.cc:491] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:02.598742 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 102797us
I20250625 13:52:02.599431 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 518) took 1024 ms. Trace:
I20250625 13:52:02.599620 1165 rpcz_store.cc:276] 0625 13:52:01.575023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.575079 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:02.562488 (+987409us) ts_tablet_manager.cc:1145] Deleting tablet 055f88e118454e32baa578cae9aceb0c
0625 13:52:02.562499 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.581744 (+ 19245us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.592896 (+ 11152us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.599203 (+ 6307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":987168}
I20250625 13:52:02.599905 1757 raft_consensus.cc:513] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.600234 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 519) took 1015 ms. Trace:
I20250625 13:52:02.600258 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 055f88e118454e32baa578cae9aceb0c (table default.Table_df6a42eba40144cf94d09e54694a85b9 [id=e50f8ace632e4ea284d17b85f474cef5]) successfully deleted
I20250625 13:52:02.600338 1165 rpcz_store.cc:276] 0625 13:52:01.584763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.584822 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:02.600040 (+1015218us) ts_tablet_manager.cc:1145] Deleting tablet 4d2d6f05348e43339e7167a9cddca2ba
0625 13:52:02.600050 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.600172 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015009}
I20250625 13:52:02.600879 1757 leader_election.cc:290] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.601681 1165 tablet_replica.cc:331] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:52:02.602041 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4d2d6f05348e43339e7167a9cddca2ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2d6f05348e43339e7167a9cddca2ba already in progress: creating tablet
W20250625 13:52:02.602308 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ae4a105e80c1460cbb21e1bdb9b356b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae4a105e80c1460cbb21e1bdb9b356b1 already in progress: creating tablet
I20250625 13:52:02.602352 1165 raft_consensus.cc:2241] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:02.602506 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d85cdadfd694486590092a7b6f61f44c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d85cdadfd694486590092a7b6f61f44c already in progress: creating tablet
W20250625 13:52:02.602720 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4ae5f1d414794474b2178b431cbc1d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ae5f1d414794474b2178b431cbc1d44 already in progress: creating tablet
W20250625 13:52:02.602906 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 17b172c7631048ceb97ccbea896f7f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17b172c7631048ceb97ccbea896f7f5b already in progress: creating tablet
I20250625 13:52:02.602991 1165 raft_consensus.cc:2270] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:02.603132 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a29708fb688a4a65b84ae2564a040b4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a29708fb688a4a65b84ae2564a040b4c already in progress: creating tablet
I20250625 13:52:02.604532 1165 ts_tablet_manager.cc:1905] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.605731 1168 raft_consensus.cc:2270] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.606120 1166 tablet_bootstrap.cc:492] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:02.607532 1166 ts_tablet_manager.cc:1397] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.010s
I20250625 13:52:02.607604 1510 leader_election.cc:290] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:02.609469 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eabd67b9ceb74bd1a5f4f115c7db0e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.610618 1168 ts_tablet_manager.cc:1905] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.610044 1166 raft_consensus.cc:357] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.611860 1166 raft_consensus.cc:383] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:02.612354 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19017cc372af43b2b5074a8990375ede" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:02.612970 851 leader_election.cc:343] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7d361e73fe3d4ea2bb6735f814345ebb
W20250625 13:52:02.613044 711 leader_election.cc:343] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:02.613210 851 leader_election.cc:343] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d0c5bea3b6214fa8af2792c5e9c411d9
W20250625 13:52:02.613361 1171 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.537s user 0.106s sys 0.291s
I20250625 13:52:02.612223 1166 raft_consensus.cc:738] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:02.615206 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3dfb86bb8bdc4e4d80cee8e155d63353 has no consensus state
I20250625 13:52:02.617789 1510 raft_consensus.cc:491] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.620890 1510 raft_consensus.cc:513] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.624120 758 tablet_service.cc:1468] Processing CreateTablet for tablet e7070fcfd40e40bfb1908b4c4013c842 (DEFAULT_TABLE table=default.Table_1480ea8e34de4a38b7f925b0079ac328 [id=a0e81714f733485caea603a5fe587926]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:02.624166 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 246091us
I20250625 13:52:02.624696 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7070fcfd40e40bfb1908b4c4013c842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.625578 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39540b81fd4746858e65be49ded0e1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.625780 750 tablet_service.cc:1468] Processing CreateTablet for tablet 575043022b1b4ca686775b2aae41a037 (DEFAULT_TABLE table=default.Table_d1099e25efdb46c084ac0792c48cc54a [id=b1154446de304a88924b0605fb75d2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.626011 1165 ts_tablet_manager.cc:1918] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:02.626205 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575043022b1b4ca686775b2aae41a037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.626284 1165 log.cc:1199] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/46316eb725bc47e695af476c0b185504
I20250625 13:52:02.628175 1510 leader_election.cc:290] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:02.629618 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89f9ffb5e4643f8915e9fd69dd3e842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.631362 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db450dc63bd4b6caf074c67490cd175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.633533 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f73f420a4d7a438eadde3d6af64f0a11" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:02.633754 913 raft_consensus.cc:2466] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:02.634279 1171 tablet_bootstrap.cc:492] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:02.634173 884 tablet_service.cc:1468] Processing CreateTablet for tablet e7070fcfd40e40bfb1908b4c4013c842 (DEFAULT_TABLE table=default.Table_1480ea8e34de4a38b7f925b0079ac328 [id=a0e81714f733485caea603a5fe587926]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.634685 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1caed92374d04bc3a365d46eb965441a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.635257 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1caed92374d04bc3a365d46eb965441a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.635848 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2c8cf5d6cd624a1797486c69bd2098fd
I20250625 13:52:02.635944 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 57afeff143c846289c6159974b00d7ba
I20250625 13:52:02.634619 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_003becbbf12b448b8d9581a65994b657"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.636997 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_003becbbf12b448b8d9581a65994b657 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.637760 1012 tablet_service.cc:1468] Processing CreateTablet for tablet fd4142572b344456a0d8248be914ad8f (DEFAULT_TABLE table=default.Table_845a00e4f2b34e549af0d49c36c7899d [id=33400d976688476eb4450525402f12be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.625922 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f033bfa4925b4d1c971789eb77efd92c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.639871 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c8cf5d6cd624a1797486c69bd2098fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:02.640491 1170 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.515s user 0.007s sys 0.043s
I20250625 13:52:02.641566 1170 tablet_bootstrap.cc:492] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:02.642031 1710 log.cc:1199] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/41a7862601f04fd7a938f7e5d5a4c0f9
I20250625 13:52:02.642666 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 57afeff143c846289c6159974b00d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.642885 1171 ts_tablet_manager.cc:1397] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.587s user 0.107s sys 0.292s
I20250625 13:52:02.645201 1171 raft_consensus.cc:357] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.645727 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755a542d04dd44a4948c6657b65ce443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.646059 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 39540b81fd4746858e65be49ded0e1df
I20250625 13:52:02.646066 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7070fcfd40e40bfb1908b4c4013c842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.646083 1171 raft_consensus.cc:383] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:02.646818 1171 raft_consensus.cc:738] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:02.648303 1710 ts_tablet_manager.cc:1939] T 41a7862601f04fd7a938f7e5d5a4c0f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:02.650540 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 39540b81fd4746858e65be49ded0e1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.651358 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5f921e5e270c4b868cc0c36c900919d2" } modify_external_catalogs: true
I20250625 13:52:02.651551 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5f921e5e270c4b868cc0c36c900919d2" } modify_external_catalogs: true
I20250625 13:52:02.653127 1020 tablet_service.cc:1468] Processing CreateTablet for tablet e7070fcfd40e40bfb1908b4c4013c842 (DEFAULT_TABLE table=default.Table_1480ea8e34de4a38b7f925b0079ac328 [id=a0e81714f733485caea603a5fe587926]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.654358 1518 raft_consensus.cc:513] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.655089 1518 leader_election.cc:290] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.657233 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ff3b675aba42adb035ff5d1404a889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.659348 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6771d8e180fe48569a0e016db0cc9b51" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.661512 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39540b81fd4746858e65be49ded0e1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.662273 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ff3b675aba42adb035ff5d1404a889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.662668 1171 consensus_queue.cc:260] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.663911 1170 ts_tablet_manager.cc:1397] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.736s user 0.008s sys 0.045s
I20250625 13:52:02.664036 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7c3f0ca6fa5482db7f57143911c838f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.664386 1765 raft_consensus.cc:491] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.664609 1765 raft_consensus.cc:513] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.664757 1170 raft_consensus.cc:357] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.664952 1170 raft_consensus.cc:383] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:02.665037 1170 raft_consensus.cc:738] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:02.665268 1170 consensus_queue.cc:260] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.665382 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 60ef50a3624b4c79acc803ff0e2ed364
I20250625 13:52:02.665786 1170 ts_tablet_manager.cc:1428] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:02.665916 983 leader_election.cc:304] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:02.668889 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412fabb4a60647e7992756879596e76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.668812 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c5bea3b6214fa8af2792c5e9c411d9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.669481 1045 raft_consensus.cc:2466] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:02.670950 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4142572b344456a0d8248be914ad8f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.676648 849 consensus_peers.cc:487] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02f2a2b595bc43309a29b3f1bc5d3766. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:02.677927 849 leader_election.cc:304] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:02.678848 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 57afeff143c846289c6159974b00d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.679227 1767 raft_consensus.cc:2802] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:02.679410 1767 raft_consensus.cc:491] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:02.679548 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ce31199dff4300a63930b1c8e6dc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.680377 1767 raft_consensus.cc:3058] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:02.680786 1171 ts_tablet_manager.cc:1428] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250625 13:52:02.681951 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eabd67b9ceb74bd1a5f4f115c7db0e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.682370 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19017cc372af43b2b5074a8990375ede" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.682761 1518 raft_consensus.cc:2802] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:02.682864 1518 raft_consensus.cc:491] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:02.682978 1518 raft_consensus.cc:3058] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:02.683118 1171 tablet_bootstrap.cc:492] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:02.684486 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a6ff3b675aba42adb035ff5d1404a889 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.684859 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 575043022b1b4ca686775b2aae41a037 (DEFAULT_TABLE table=default.Table_d1099e25efdb46c084ac0792c48cc54a [id=b1154446de304a88924b0605fb75d2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.685123 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.682382 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755a542d04dd44a4948c6657b65ce443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.684684 1765 leader_election.cc:290] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.680050 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "643928ca4fc94ce58431a0525f2abb99" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.686740 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ce31199dff4300a63930b1c8e6dc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.669790 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d361e73fe3d4ea2bb6735f814345ebb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.690282 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:02.691426 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a82adc5aa045f49e092dababafe4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.692173 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 39540b81fd4746858e65be49ded0e1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:02.692498 892 tablet_service.cc:1468] Processing CreateTablet for tablet 575043022b1b4ca686775b2aae41a037 (DEFAULT_TABLE table=default.Table_d1099e25efdb46c084ac0792c48cc54a [id=b1154446de304a88924b0605fb75d2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.694723 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 41a7862601f04fd7a938f7e5d5a4c0f9 (table default.Table_478b0037c80e40048d4f8ebc92748f34 [id=13d38d05aad34e22884b9e07610b0e87]) successfully deleted
W20250625 13:52:02.692301 849 leader_election.cc:343] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.696082 1165 ts_tablet_manager.cc:1939] T 46316eb725bc47e695af476c0b185504 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:02.698149 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85fa343e30e4e838ab81cd775c298ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.698118 851 leader_election.cc:343] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 643928ca4fc94ce58431a0525f2abb99
I20250625 13:52:02.698349 851 leader_election.cc:304] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:02.700245 711 leader_election.cc:343] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.700402 711 leader_election.cc:304] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:02.701236 711 raft_consensus.cc:2684] T f033bfa4925b4d1c971789eb77efd92c P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 13:52:02.701960 711 leader_election.cc:343] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.702381 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575043022b1b4ca686775b2aae41a037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.702435 1748 raft_consensus.cc:2747] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.702692 711 leader_election.cc:304] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:02.703109 711 leader_election.cc:343] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.703905 711 leader_election.cc:304] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:02.704322 711 leader_election.cc:343] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.704466 711 leader_election.cc:304] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:02.704839 1669 raft_consensus.cc:2747] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.705034 1669 raft_consensus.cc:2747] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.705168 1757 raft_consensus.cc:2747] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.707966 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.708063 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 57afeff143c846289c6159974b00d7ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.708246 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 39540b81fd4746858e65be49ded0e1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.708492 896 tablet_service.cc:1468] Processing CreateTablet for tablet 4b61d4f7f0474aac9b46c532f6f403a0 (DEFAULT_TABLE table=default.Table_0ff1884a92fe4a4490d4a3f5594fa263 [id=e819735fd2f5444cae36220b2c1d8b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.708590 744 tablet_service.cc:1468] Processing CreateTablet for tablet 68a7712455694a1cbead413230620827 (DEFAULT_TABLE table=default.Table_f5d01b07d90441a79d2ccd6c8fa92610 [id=1e19c242b306485dbce55fdc99fe5720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.708940 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a7712455694a1cbead413230620827. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.709442 880 tablet_service.cc:1468] Processing CreateTablet for tablet 8b0e49ca4dee42c59c0a2927a47aa320 (DEFAULT_TABLE table=default.Table_20227b2471d345dc90782795a2d1de32 [id=bc1d4ec693154bb5aa80709fed7d41e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.710034 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 (DEFAULT_TABLE table=default.Table_7c6963b0fa6a4bce977904672ec5ae28 [id=0758bd83d4514fa8960a40e865ad1633]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.710395 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.711251 754 tablet_service.cc:1468] Processing CreateTablet for tablet 4b61d4f7f0474aac9b46c532f6f403a0 (DEFAULT_TABLE table=default.Table_0ff1884a92fe4a4490d4a3f5594fa263 [id=e819735fd2f5444cae36220b2c1d8b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.711661 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b61d4f7f0474aac9b46c532f6f403a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.712975 886 tablet_service.cc:1468] Processing CreateTablet for tablet d0baddb123f0401a98662b4b483b9587 (DEFAULT_TABLE table=default.Table_7700dabe9c4446908a9535b1b946a3e0 [id=d70f5f4396a74c358b3fbf3a9dc12489]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.713208 877 tablet_service.cc:1468] Processing CreateTablet for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 (DEFAULT_TABLE table=default.Table_7c6963b0fa6a4bce977904672ec5ae28 [id=0758bd83d4514fa8960a40e865ad1633]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.713482 879 tablet_service.cc:1468] Processing CreateTablet for tablet 57afeff143c846289c6159974b00d7ba (DEFAULT_TABLE table=default.Table_4dd664d07c7f4b22a44352da05e41e94 [id=4bd340b694ad46c3857db6abc831b9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.713652 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b61d4f7f0474aac9b46c532f6f403a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.714206 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0e49ca4dee42c59c0a2927a47aa320. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.716004 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 68a7712455694a1cbead413230620827 (DEFAULT_TABLE table=default.Table_f5d01b07d90441a79d2ccd6c8fa92610 [id=1e19c242b306485dbce55fdc99fe5720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.690310 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c8cf5d6cd624a1797486c69bd2098fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:02.717653 985 leader_election.cc:343] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d0c5bea3b6214fa8af2792c5e9c411d9
W20250625 13:52:02.717976 985 leader_election.cc:343] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cf1664afbab647b3baeed6391196bed9
I20250625 13:52:02.720196 1767 raft_consensus.cc:513] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.721714 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd4142572b344456a0d8248be914ad8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.721690 882 tablet_service.cc:1468] Processing CreateTablet for tablet 68a7712455694a1cbead413230620827 (DEFAULT_TABLE table=default.Table_f5d01b07d90441a79d2ccd6c8fa92610 [id=1e19c242b306485dbce55fdc99fe5720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.723484 1710 tablet_replica.cc:331] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:02.725028 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0baddb123f0401a98662b4b483b9587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.726884 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7070fcfd40e40bfb1908b4c4013c842. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.727434 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a85fa343e30e4e838ab81cd775c298ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.731643 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57afeff143c846289c6159974b00d7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.731987 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.733533 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 57afeff143c846289c6159974b00d7ba (DEFAULT_TABLE table=default.Table_4dd664d07c7f4b22a44352da05e41e94 [id=4bd340b694ad46c3857db6abc831b9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.734138 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 8b0e49ca4dee42c59c0a2927a47aa320 (DEFAULT_TABLE table=default.Table_20227b2471d345dc90782795a2d1de32 [id=bc1d4ec693154bb5aa80709fed7d41e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.734730 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d0baddb123f0401a98662b4b483b9587 (DEFAULT_TABLE table=default.Table_7700dabe9c4446908a9535b1b946a3e0 [id=d70f5f4396a74c358b3fbf3a9dc12489]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.736250 1767 leader_election.cc:290] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:02.737138 749 tablet_service.cc:1468] Processing CreateTablet for tablet d0baddb123f0401a98662b4b483b9587 (DEFAULT_TABLE table=default.Table_7700dabe9c4446908a9535b1b946a3e0 [id=d70f5f4396a74c358b3fbf3a9dc12489]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.737681 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0baddb123f0401a98662b4b483b9587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.738863 743 tablet_service.cc:1468] Processing CreateTablet for tablet 57afeff143c846289c6159974b00d7ba (DEFAULT_TABLE table=default.Table_4dd664d07c7f4b22a44352da05e41e94 [id=4bd340b694ad46c3857db6abc831b9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.739288 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57afeff143c846289c6159974b00d7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.741129 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a7712455694a1cbead413230620827. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.741520 753 tablet_service.cc:1468] Processing CreateTablet for tablet 8b0e49ca4dee42c59c0a2927a47aa320 (DEFAULT_TABLE table=default.Table_20227b2471d345dc90782795a2d1de32 [id=bc1d4ec693154bb5aa80709fed7d41e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.742106 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0e49ca4dee42c59c0a2927a47aa320. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.614641 1166 consensus_queue.cc:260] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.749122 755 tablet_service.cc:1468] Processing CreateTablet for tablet 54ea4c4d7756433d85fe9b816b4816ae (DEFAULT_TABLE table=default.Table_7e1bff138de841eda69caac9be5438bb [id=8ca777c8ef074fb9b4db9d305e34062e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.750571 746 tablet_service.cc:1468] Processing CreateTablet for tablet 7ea504789a27455e81bb14d829f444ba (DEFAULT_TABLE table=default.Table_e1a2fad7ba6143feb9d139ab60083e8a [id=169f630cf57647fe85bf3579a3c83667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.754981 1518 raft_consensus.cc:513] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.767161 752 tablet_service.cc:1468] Processing CreateTablet for tablet beab7389141b49ef90f07e3264091ac3 (DEFAULT_TABLE table=default.Table_6515cc13bb054b8d89752a85d63f6996 [id=a297be70f6f64fd791ac3b874ffe01ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.768465 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68a7712455694a1cbead413230620827. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.770390 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ea4c4d7756433d85fe9b816b4816ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.772612 1767 raft_consensus.cc:491] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.775463 757 tablet_service.cc:1468] Processing CreateTablet for tablet daa16c7ca9c7480dbd5fbd8114a449f2 (DEFAULT_TABLE table=default.Table_2fb3cd371d784e4c980b6f63c84d2f09 [id=0faa5548640c482eb64435ce36b7c835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.770413 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 4b61d4f7f0474aac9b46c532f6f403a0 (DEFAULT_TABLE table=default.Table_0ff1884a92fe4a4490d4a3f5594fa263 [id=e819735fd2f5444cae36220b2c1d8b18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.778540 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c8cf5d6cd624a1797486c69bd2098fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:02.781473 747 tablet_service.cc:1468] Processing CreateTablet for tablet 9aadc67058554e87bb58d14b911333c4 (DEFAULT_TABLE table=default.Table_4d8e666a80a345f7ba2ad05b84ce5a05 [id=560d105a82934a11a344edfee582205c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.782418 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea504789a27455e81bb14d829f444ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.784549 1710 raft_consensus.cc:2241] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.785038 751 tablet_service.cc:1468] Processing CreateTablet for tablet 43941422259b4413bc71acfce8928e1e (DEFAULT_TABLE table=default.Table_194c4cbe54834768a869225974bea8ab [id=58e6162e8e124b90a20a9fe9a0f011af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.785341 745 tablet_service.cc:1468] Processing CreateTablet for tablet aef43c4962c747dbb43039d1e706bbd9 (DEFAULT_TABLE table=default.Table_7e64f15192894423be9dcd6d12afcc67 [id=bba3ab683b4947b0a956e9036ad24120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.785694 1011 tablet_service.cc:1468] Processing CreateTablet for tablet beab7389141b49ef90f07e3264091ac3 (DEFAULT_TABLE table=default.Table_6515cc13bb054b8d89752a85d63f6996 [id=a297be70f6f64fd791ac3b874ffe01ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.786422 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575043022b1b4ca686775b2aae41a037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.788280 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aadc67058554e87bb58d14b911333c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.789162 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd4f0857f52449e1b26fde09c6c7c9d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:02.791368 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c5bea3b6214fa8af2792c5e9c411d9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:02.793123 740 tablet_service.cc:1468] Processing CreateTablet for tablet 3c6dd5b87ff34151836a3c29740e7c11 (DEFAULT_TABLE table=default.Table_796954f1442545d2aa7d7ffaf74e29d2 [id=b42de9585c1a4dfc81f3d613df93b089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.793447 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef43c4962c747dbb43039d1e706bbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.794528 1044 raft_consensus.cc:3058] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:02.797601 985 leader_election.cc:343] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f73f420a4d7a438eadde3d6af64f0a11
I20250625 13:52:02.798611 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 (DEFAULT_TABLE table=default.Table_7c6963b0fa6a4bce977904672ec5ae28 [id=0758bd83d4514fa8960a40e865ad1633]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.799822 893 tablet_service.cc:1468] Processing CreateTablet for tablet 54ea4c4d7756433d85fe9b816b4816ae (DEFAULT_TABLE table=default.Table_7e1bff138de841eda69caac9be5438bb [id=8ca777c8ef074fb9b4db9d305e34062e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.799898 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 54ea4c4d7756433d85fe9b816b4816ae (DEFAULT_TABLE table=default.Table_7e1bff138de841eda69caac9be5438bb [id=8ca777c8ef074fb9b4db9d305e34062e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.801180 1171 tablet_bootstrap.cc:654] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:02.801373 750 tablet_service.cc:1468] Processing CreateTablet for tablet e29bade90b184d2483cb1fe85033a03c (DEFAULT_TABLE table=default.Table_e90d64fdf05e442aa9da241d5fa1263b [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.802505 741 tablet_service.cc:1468] Processing CreateTablet for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 (DEFAULT_TABLE table=default.Table_f7ba037477124796ab2a66a351968c19 [id=2fe7121baaa140ca8670b9e39e572fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.804462 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6dd5b87ff34151836a3c29740e7c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.813935 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43941422259b4413bc71acfce8928e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.817081 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa16c7ca9c7480dbd5fbd8114a449f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.822492 895 tablet_service.cc:1468] Processing CreateTablet for tablet 43941422259b4413bc71acfce8928e1e (DEFAULT_TABLE table=default.Table_194c4cbe54834768a869225974bea8ab [id=58e6162e8e124b90a20a9fe9a0f011af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.822660 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beab7389141b49ef90f07e3264091ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.826216 887 tablet_service.cc:1468] Processing CreateTablet for tablet 9aadc67058554e87bb58d14b911333c4 (DEFAULT_TABLE table=default.Table_4d8e666a80a345f7ba2ad05b84ce5a05 [id=560d105a82934a11a344edfee582205c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.826934 880 tablet_service.cc:1468] Processing CreateTablet for tablet aef43c4962c747dbb43039d1e706bbd9 (DEFAULT_TABLE table=default.Table_7e64f15192894423be9dcd6d12afcc67 [id=bba3ab683b4947b0a956e9036ad24120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.835173 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b61d4f7f0474aac9b46c532f6f403a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.836431 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beab7389141b49ef90f07e3264091ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.837255 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ea4c4d7756433d85fe9b816b4816ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.839226 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.840487 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0baddb123f0401a98662b4b483b9587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.841311 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0e49ca4dee42c59c0a2927a47aa320. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.842613 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57afeff143c846289c6159974b00d7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.842867 748 tablet_service.cc:1468] Processing CreateTablet for tablet 47eb205a9bcd4608930d9355978a44ce (DEFAULT_TABLE table=default.Table_0f82cde67a844d2dadb19c95b952db96 [id=7d3b33fe20ec45049b6e76e4889a1f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.843427 1518 leader_election.cc:290] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:02.843858 1710 raft_consensus.cc:2270] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.844558 1710 ts_tablet_manager.cc:1905] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.845849 742 tablet_service.cc:1468] Processing CreateTablet for tablet af934858a4af4621bb55eb5a528fa14c (DEFAULT_TABLE table=default.Table_f89c8b7c36634729a135e0a425694693 [id=6dd4f89ba83a438d8afaf7b242f315fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.846239 1044 raft_consensus.cc:2466] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:02.850651 1170 tablet_bootstrap.cc:492] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:02.850793 1748 raft_consensus.cc:491] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c1aa07ab6bc4b0089e46951240a37c9)
I20250625 13:52:02.852600 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29bade90b184d2483cb1fe85033a03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.853787 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af934858a4af4621bb55eb5a528fa14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.859108 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05364e23e0ec4cffb92a8df2d38ca4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.865255 739 tablet_service.cc:1468] Processing CreateTablet for tablet f197ea67c34546a7981bcd9df94702e6 (DEFAULT_TABLE table=default.Table_09dd798f73ca46cf9edc4fd651169efd [id=b3cf26be0057490b980afb2e0154277c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.874384 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2c8cf5d6cd624a1797486c69bd2098fd (DEFAULT_TABLE table=default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.876225 1767 raft_consensus.cc:513] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.877107 1767 leader_election.cc:290] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:02.881148 1165 tablet_replica.cc:331] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:02.881752 1165 raft_consensus.cc:2241] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:02.882115 1165 raft_consensus.cc:2270] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:02.883782 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 43941422259b4413bc71acfce8928e1e (DEFAULT_TABLE table=default.Table_194c4cbe54834768a869225974bea8ab [id=58e6162e8e124b90a20a9fe9a0f011af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.883843 1165 ts_tablet_manager.cc:1905] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:02.884287 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47eb205a9bcd4608930d9355978a44ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.888052 878 tablet_service.cc:1468] Processing CreateTablet for tablet beab7389141b49ef90f07e3264091ac3 (DEFAULT_TABLE table=default.Table_6515cc13bb054b8d89752a85d63f6996 [id=a297be70f6f64fd791ac3b874ffe01ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.888936 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54ea4c4d7756433d85fe9b816b4816ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.889984 1168 ts_tablet_manager.cc:1918] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:02.890825 877 tablet_service.cc:1468] Processing CreateTablet for tablet daa16c7ca9c7480dbd5fbd8114a449f2 (DEFAULT_TABLE table=default.Table_2fb3cd371d784e4c980b6f63c84d2f09 [id=0faa5548640c482eb64435ce36b7c835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.889990 1014 tablet_service.cc:1468] Processing CreateTablet for tablet daa16c7ca9c7480dbd5fbd8114a449f2 (DEFAULT_TABLE table=default.Table_2fb3cd371d784e4c980b6f63c84d2f09 [id=0faa5548640c482eb64435ce36b7c835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.895758 883 tablet_service.cc:1468] Processing CreateTablet for tablet 7ea504789a27455e81bb14d829f444ba (DEFAULT_TABLE table=default.Table_e1a2fad7ba6143feb9d139ab60083e8a [id=169f630cf57647fe85bf3579a3c83667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.897924 1168 log.cc:1199] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/1f6be7189f8f45ccb329ef5d2fd8da8d
I20250625 13:52:02.898294 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f197ea67c34546a7981bcd9df94702e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.900991 1748 raft_consensus.cc:513] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:02.901751 1748 leader_election.cc:290] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:02.904114 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa16c7ca9c7480dbd5fbd8114a449f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.904825 849 leader_election.cc:343] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.905658 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43941422259b4413bc71acfce8928e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.906466 882 tablet_service.cc:1468] Processing CreateTablet for tablet e29bade90b184d2483cb1fe85033a03c (DEFAULT_TABLE table=default.Table_e90d64fdf05e442aa9da241d5fa1263b [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.899158 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 7ea504789a27455e81bb14d829f444ba (DEFAULT_TABLE table=default.Table_e1a2fad7ba6143feb9d139ab60083e8a [id=169f630cf57647fe85bf3579a3c83667]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:02.907157 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 202822us
I20250625 13:52:02.907163 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3c6dd5b87ff34151836a3c29740e7c11 (DEFAULT_TABLE table=default.Table_796954f1442545d2aa7d7ffaf74e29d2 [id=b42de9585c1a4dfc81f3d613df93b089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.907855 891 tablet_service.cc:1468] Processing CreateTablet for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 (DEFAULT_TABLE table=default.Table_f7ba037477124796ab2a66a351968c19 [id=2fe7121baaa140ca8670b9e39e572fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.908922 888 tablet_service.cc:1468] Processing CreateTablet for tablet 47eb205a9bcd4608930d9355978a44ce (DEFAULT_TABLE table=default.Table_0f82cde67a844d2dadb19c95b952db96 [id=7d3b33fe20ec45049b6e76e4889a1f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.909636 884 tablet_service.cc:1468] Processing CreateTablet for tablet af934858a4af4621bb55eb5a528fa14c (DEFAULT_TABLE table=default.Table_f89c8b7c36634729a135e0a425694693 [id=6dd4f89ba83a438d8afaf7b242f315fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.911336 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa16c7ca9c7480dbd5fbd8114a449f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.911952 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea504789a27455e81bb14d829f444ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.913990 746 tablet_service.cc:1468] Processing CreateTablet for tablet c431abffd05c4136b7510c60eee2a8b1 (DEFAULT_TABLE table=default.Table_d18abaf5b02340fa82d3fee69699e1e3 [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.915498 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beab7389141b49ef90f07e3264091ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.916839 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43941422259b4413bc71acfce8928e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.917268 747 tablet_service.cc:1468] Processing CreateTablet for tablet 68831b9ad1c14f5e9c954cbd06094896 (DEFAULT_TABLE table=default.Table_4dc8c488c3e44f3f882049d1c0f8feb2 [id=0532faedda28481f9742008aecb6bfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.918007 743 tablet_service.cc:1468] Processing CreateTablet for tablet d2f545a5934748b2af767d37ea937fca (DEFAULT_TABLE table=default.Table_5f921e5e270c4b868cc0c36c900919d2 [id=612d50114aac44d1ae69ac99b3c0dab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.922637 849 leader_election.cc:304] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:02.923738 894 tablet_service.cc:1468] Processing CreateTablet for tablet 2c8cf5d6cd624a1797486c69bd2098fd (DEFAULT_TABLE table=default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.927959 749 tablet_service.cc:1468] Processing CreateTablet for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 (DEFAULT_TABLE table=default.Table_d8fbb4df3d0745b6a9671d0dbfb1c789 [id=57a1163f204646f6acf00157f75e5209]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.930943 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8cf5d6cd624a1797486c69bd2098fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.934226 745 tablet_service.cc:1468] Processing CreateTablet for tablet 5d478ac74f4f46c6a29a6ac031152498 (DEFAULT_TABLE table=default.Table_a53e9e99051944a1bc66244b53112731 [id=b650eb8998664ad593aba664d70d1898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.925897 890 tablet_service.cc:1468] Processing CreateTablet for tablet 5d478ac74f4f46c6a29a6ac031152498 (DEFAULT_TABLE table=default.Table_a53e9e99051944a1bc66244b53112731 [id=b650eb8998664ad593aba664d70d1898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.940668 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 9aadc67058554e87bb58d14b911333c4 (DEFAULT_TABLE table=default.Table_4d8e666a80a345f7ba2ad05b84ce5a05 [id=560d105a82934a11a344edfee582205c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.944206 896 tablet_service.cc:1468] Processing CreateTablet for tablet c431abffd05c4136b7510c60eee2a8b1 (DEFAULT_TABLE table=default.Table_d18abaf5b02340fa82d3fee69699e1e3 [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.945019 756 tablet_service.cc:1468] Processing CreateTablet for tablet e438e46874f841d2ac972b953ddb4608 (DEFAULT_TABLE table=default.Table_0ca3566f743046db9a91f2a4040f2e03 [id=09abf01183f64858a59a83bf152b107b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.944543 1776 raft_consensus.cc:491] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:02.949393 740 tablet_service.cc:1468] Processing CreateTablet for tablet bd4cc91d274a45a393beffaaa59a35f0 (DEFAULT_TABLE table=default.Table_88d5d2614816414f8a45b3526e1608d8 [id=677c08587cdd48119667df6818cc2dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.951387 1166 ts_tablet_manager.cc:1428] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.301s user 0.002s sys 0.001s
I20250625 13:52:02.952224 893 tablet_service.cc:1468] Processing CreateTablet for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 (DEFAULT_TABLE table=default.Table_d8fbb4df3d0745b6a9671d0dbfb1c789 [id=57a1163f204646f6acf00157f75e5209]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.952566 1171 tablet_bootstrap.cc:492] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:02.952629 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aadc67058554e87bb58d14b911333c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.952333 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.952730 1171 ts_tablet_manager.cc:1397] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.272s user 0.004s sys 0.017s
I20250625 13:52:02.953639 1171 raft_consensus.cc:357] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.953840 1171 raft_consensus.cc:383] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:02.953938 1171 raft_consensus.cc:738] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:02.954180 1171 consensus_queue.cc:260] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.954443 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68831b9ad1c14f5e9c954cbd06094896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.954833 1171 ts_tablet_manager.cc:1428] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:02.955267 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8cf5d6cd624a1797486c69bd2098fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.955430 892 tablet_service.cc:1468] Processing CreateTablet for tablet f197ea67c34546a7981bcd9df94702e6 (DEFAULT_TABLE table=default.Table_09dd798f73ca46cf9edc4fd651169efd [id=b3cf26be0057490b980afb2e0154277c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.955842 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f197ea67c34546a7981bcd9df94702e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.957541 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d478ac74f4f46c6a29a6ac031152498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.959542 1168 ts_tablet_manager.cc:1939] T 1f6be7189f8f45ccb329ef5d2fd8da8d P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:02.959826 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29bade90b184d2483cb1fe85033a03c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.959784 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 179746us
I20250625 13:52:02.960747 1748 raft_consensus.cc:2747] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:02.961215 849 leader_election.cc:343] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:02.961374 849 leader_election.cc:304] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:02.961701 1767 raft_consensus.cc:2747] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:02.961812 849 leader_election.cc:304] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:02.962137 1767 raft_consensus.cc:2802] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:02.962280 1767 raft_consensus.cc:695] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:02.962540 1767 consensus_queue.cc:237] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.962929 881 tablet_service.cc:1468] Processing CreateTablet for tablet d2f545a5934748b2af767d37ea937fca (DEFAULT_TABLE table=default.Table_5f921e5e270c4b868cc0c36c900919d2 [id=612d50114aac44d1ae69ac99b3c0dab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.963323 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2f545a5934748b2af767d37ea937fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.963542 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef43c4962c747dbb43039d1e706bbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.963652 879 tablet_service.cc:1468] Processing CreateTablet for tablet bd4cc91d274a45a393beffaaa59a35f0 (DEFAULT_TABLE table=default.Table_88d5d2614816414f8a45b3526e1608d8 [id=677c08587cdd48119667df6818cc2dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.964058 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4cc91d274a45a393beffaaa59a35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.967370 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6dd5b87ff34151836a3c29740e7c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.968142 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d478ac74f4f46c6a29a6ac031152498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.968875 1166 tablet_bootstrap.cc:492] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:02.971417 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c431abffd05c4136b7510c60eee2a8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.971649 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af934858a4af4621bb55eb5a528fa14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.974985 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47eb205a9bcd4608930d9355978a44ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.975667 849 leader_election.cc:343] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: bd5ba76c130a4711af5b21e9eb8562e7
I20250625 13:52:02.976233 1770 consensus_queue.cc:1035] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:02.976737 1770 consensus_queue.cc:1035] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:02.978976 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05364e23e0ec4cffb92a8df2d38ca4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:02.975107 1012 tablet_service.cc:1468] Processing CreateTablet for tablet e29bade90b184d2483cb1fe85033a03c (DEFAULT_TABLE table=default.Table_e90d64fdf05e442aa9da241d5fa1263b [id=fd00675638fa4c52b329d41cc4fa2861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.980103 1770 raft_consensus.cc:491] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c1aa07ab6bc4b0089e46951240a37c9)
I20250625 13:52:02.980244 1770 raft_consensus.cc:513] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.981036 1770 leader_election.cc:290] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:02.982340 1013 tablet_service.cc:1468] Processing CreateTablet for tablet aef43c4962c747dbb43039d1e706bbd9 (DEFAULT_TABLE table=default.Table_7e64f15192894423be9dcd6d12afcc67 [id=bba3ab683b4947b0a956e9036ad24120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.963972 1776 raft_consensus.cc:513] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:02.982885 757 tablet_service.cc:1468] Processing CreateTablet for tablet bf7463962f39468c85fb3dc2edcf608f (DEFAULT_TABLE table=default.Table_45ffe0341a354703ab5ab6c66bd27f18 [id=772bd28ed2964d3b9d09189c07cc34f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.983609 889 tablet_service.cc:1468] Processing CreateTablet for tablet e438e46874f841d2ac972b953ddb4608 (DEFAULT_TABLE table=default.Table_0ca3566f743046db9a91f2a4040f2e03 [id=09abf01183f64858a59a83bf152b107b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.984027 885 tablet_service.cc:1468] Processing CreateTablet for tablet bf7463962f39468c85fb3dc2edcf608f (DEFAULT_TABLE table=default.Table_45ffe0341a354703ab5ab6c66bd27f18 [id=772bd28ed2964d3b9d09189c07cc34f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:02.985355 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_97642fc925bc492e9be22bec273710f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.986095 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97642fc925bc492e9be22bec273710f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.987524 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2a0fc7dfe9884d9eb17952b2d9d3d5ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:02.988989 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e438e46874f841d2ac972b953ddb4608. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:02.989001 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a0fc7dfe9884d9eb17952b2d9d3d5ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.996101 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2c513d91226c4bf6aa6643cc25131331"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.996851 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c513d91226c4bf6aa6643cc25131331 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:02.997967 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7e64f15192894423be9dcd6d12afcc67" } new_table_name: "default.Table_bd752418b476484dbe546a31f63ec6d9" modify_external_catalogs: true
I20250625 13:52:02.998370 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3034d62d25c74b3a9d694ce4d7ba717c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:02.999706 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3034d62d25c74b3a9d694ce4d7ba717c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.001483 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d2f545a5934748b2af767d37ea937fca
I20250625 13:52:03.002905 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea504789a27455e81bb14d829f444ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.004087 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_84313e8427b643219197cc4ad05fca40" } new_table_name: "default.Table_49756985b3fc428cbc871659e5aabdc6" modify_external_catalogs: true
I20250625 13:52:03.004916 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1f52a2da8b374d99952264989d3f5b90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.006263 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f52a2da8b374d99952264989d3f5b90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.005538 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b507798c190f493d9c549c62c02bc868" } new_table_name: "default.Table_49a5b264518840179315082ba4922b6c" modify_external_catalogs: true
I20250625 13:52:03.007290 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f6b4ad595d094974adb62da4fe761a46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.007848 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6b4ad595d094974adb62da4fe761a46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.008555 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7c37f5cca4df4a929018f7a81e5cdcb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.008930 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf1664afbab647b3baeed6391196bed9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:03.009073 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c37f5cca4df4a929018f7a81e5cdcb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.016383 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6df3ba6b40a746ea8650dae9fc9922e8" } new_table_name: "default.Table_e2f0cb92dbc846c68be94207b9a9126b" modify_external_catalogs: true
I20250625 13:52:03.024559 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_af1cd42de11e4022ac9d9550faca59b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.025188 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af1cd42de11e4022ac9d9550faca59b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.028311 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33015
--webserver_interface=127.0.104.62
--webserver_port=44303
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 13:52:03.039503 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f89c8b7c36634729a135e0a425694693" } new_table_name: "default.Table_105e4d24875c4713ae92e3577d1881dd" modify_external_catalogs: true
I20250625 13:52:03.042016 877 tablet_service.cc:1468] Processing CreateTablet for tablet 68831b9ad1c14f5e9c954cbd06094896 (DEFAULT_TABLE table=default.Table_4dc8c488c3e44f3f882049d1c0f8feb2 [id=0532faedda28481f9742008aecb6bfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.042480 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 255) took 6681 ms. Trace:
I20250625 13:52:03.043134 1171 tablet_bootstrap.cc:492] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:03.046932 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35aedb3b590d4917bac8fe0962d68fb3 consensus state has no leader
I20250625 13:52:03.055126 1168 rpcz_store.cc:276] 0625 13:51:56.320490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.320550 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:02.091601 (+5771051us) ts_tablet_manager.cc:1145] Deleting tablet 1f6be7189f8f45ccb329ef5d2fd8da8d
0625 13:52:02.091611 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.367212 (+275601us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x6160000b2c54. stack: 00007f37024dc336 00007f37024dc0f0 00007f37014d23ad 0000000000971004 0000000000970fcd 000000000097099a 00007f370df3c000 00007f370f24e093 00007f3711d2b137 00007f3711d5d1c4 00007f3711d3ec20 00007f3711d3e9ad 0000000000a7573c 00007f37025219df 00007f3702524cab 00007f3702524b28
0625 13:52:02.672425 (+305213us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:02.883427 (+211002us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.001479 (+118052us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2151,"spinlock_wait_cycles":343524224,"tablet-delete.queue_time_us":5745636}
I20250625 13:52:03.060132 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 1f6be7189f8f45ccb329ef5d2fd8da8d (table default.Table_1be3d9db61ab4b56b98b70af09df98b3 [id=68e44a6915224107b42ca15194e2fed9]) successfully deleted
I20250625 13:52:03.067617 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.067966 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7463962f39468c85fb3dc2edcf608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.071316 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e438e46874f841d2ac972b953ddb4608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.070667 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39ac79102eab46688e767faed6d6f114" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:03.071722 1168 tablet_replica.cc:331] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:03.074625 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef43c4962c747dbb43039d1e706bbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.074733 1168 raft_consensus.cc:2241] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.074914 1168 pending_rounds.cc:70] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:03.075016 1168 pending_rounds.cc:77] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520590180777984 op_type: NO_OP noop_request { }
I20250625 13:52:03.075160 1168 raft_consensus.cc:2887] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:03.075322 1168 raft_consensus.cc:2270] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.076061 1168 ts_tablet_manager.cc:1905] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.076313 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68831b9ad1c14f5e9c954cbd06094896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.077246 1049 raft_consensus.cc:2466] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:03.073467 883 tablet_service.cc:1515] Processing DeleteTablet for tablet d2f545a5934748b2af767d37ea937fca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.079084 1170 tablet_bootstrap.cc:654] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.082167 1044 raft_consensus.cc:3058] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:03.088064 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2f545a5934748b2af767d37ea937fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.087957 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 3c6dd5b87ff34151836a3c29740e7c11 (DEFAULT_TABLE table=default.Table_796954f1442545d2aa7d7ffaf74e29d2 [id=b42de9585c1a4dfc81f3d613df93b089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.095554 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4cc91d274a45a393beffaaa59a35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.096652 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39ac79102eab46688e767faed6d6f114" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:03.097232 985 consensus_peers.cc:487] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ba63184ca2b4badafcf08309ef75164. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:03.104600 849 leader_election.cc:304] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:03.109494 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 (DEFAULT_TABLE table=default.Table_f7ba037477124796ab2a66a351968c19 [id=2fe7121baaa140ca8670b9e39e572fea]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:03.111579 711 leader_election.cc:343] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:03.113462 849 leader_election.cc:343] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: d4f19ddc51324cc18242c6081f3f7de5
I20250625 13:52:03.113658 1770 raft_consensus.cc:2802] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:03.114204 1770 raft_consensus.cc:491] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:03.114326 1770 raft_consensus.cc:3058] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:03.120501 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aadc67058554e87bb58d14b911333c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.125380 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05364e23e0ec4cffb92a8df2d38ca4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.129451 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6dd5b87ff34151836a3c29740e7c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.139494 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29bade90b184d2483cb1fe85033a03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.144595 1171 tablet_bootstrap.cc:654] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.161321 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 47eb205a9bcd4608930d9355978a44ce (DEFAULT_TABLE table=default.Table_0f82cde67a844d2dadb19c95b952db96 [id=7d3b33fe20ec45049b6e76e4889a1f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.162024 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47eb205a9bcd4608930d9355978a44ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.166100 1770 raft_consensus.cc:513] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.167299 1770 leader_election.cc:290] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:03.167397 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf1664afbab647b3baeed6391196bed9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:03.167692 1047 raft_consensus.cc:3058] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:03.171096 1047 raft_consensus.cc:2466] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:03.174937 849 leader_election.cc:304] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:03.182510 1021 tablet_service.cc:1468] Processing CreateTablet for tablet af934858a4af4621bb55eb5a528fa14c (DEFAULT_TABLE table=default.Table_f89c8b7c36634729a135e0a425694693 [id=6dd4f89ba83a438d8afaf7b242f315fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.183120 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af934858a4af4621bb55eb5a528fa14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.090291 1776 leader_election.cc:290] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:03.191269 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 46316eb725bc47e695af476c0b185504 (table default.Table_7a2ca389ecd148b9be0e90c130fe6431 [id=26d0bdba8be543d89d5337c86bbf6e57]) successfully deleted
W20250625 13:52:03.191632 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e3ef7f5b474466bbc19fd31c6b6d99c already in progress: creating tablet
W20250625 13:52:03.191849 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2404e712e18146b68757bd77362c1358 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2404e712e18146b68757bd77362c1358 already in progress: creating tablet
W20250625 13:52:03.192240 851 leader_election.cc:343] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d0c5bea3b6214fa8af2792c5e9c411d9
I20250625 13:52:03.195361 1770 raft_consensus.cc:2802] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:03.195629 1770 raft_consensus.cc:695] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
W20250625 13:52:03.195745 851 consensus_peers.cc:487] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c911d9ecbded417daf70e85c7cbcd52e. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:03.196122 851 leader_election.cc:343] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cf1664afbab647b3baeed6391196bed9
W20250625 13:52:03.196557 851 leader_election.cc:343] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: bd5ba76c130a4711af5b21e9eb8562e7
I20250625 13:52:03.197099 851 leader_election.cc:304] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:03.197324 851 raft_consensus.cc:2684] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 13:52:03.198185 851 leader_election.cc:343] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d4f19ddc51324cc18242c6081f3f7de5
I20250625 13:52:03.198307 851 leader_election.cc:304] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:03.198524 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7463962f39468c85fb3dc2edcf608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.198556 1748 raft_consensus.cc:2747] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:03.200196 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c431abffd05c4136b7510c60eee2a8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.184465 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 2c8cf5d6cd624a1797486c69bd2098fd (DEFAULT_TABLE table=default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.202963 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c8cf5d6cd624a1797486c69bd2098fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.203059 1770 consensus_queue.cc:237] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:03.205506 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 120440us
I20250625 13:52:03.206755 1166 tablet_bootstrap.cc:654] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.208931 1748 raft_consensus.cc:491] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.209092 1748 raft_consensus.cc:513] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.209853 1748 leader_election.cc:290] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:03.210875 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb1c7a85da844d34be805843bc60d350" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:03.211417 711 leader_election.cc:343] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:03.211510 849 leader_election.cc:343] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.197808 1018 tablet_service.cc:1468] Processing CreateTablet for tablet f197ea67c34546a7981bcd9df94702e6 (DEFAULT_TABLE table=default.Table_09dd798f73ca46cf9edc4fd651169efd [id=b3cf26be0057490b980afb2e0154277c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.211577 711 leader_election.cc:304] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:03.211889 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f197ea67c34546a7981bcd9df94702e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.212777 1776 raft_consensus.cc:2747] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:03.216092 851 consensus_peers.cc:487] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15e58fbec94e4ff9b7ac9830b576fe7f. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:03.216423 742 tablet_service.cc:1515] Processing DeleteTablet for tablet d2f545a5934748b2af767d37ea937fca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:03.223431 851 leader_election.cc:343] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cf1664afbab647b3baeed6391196bed9
W20250625 13:52:03.223866 851 leader_election.cc:343] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: bb1c7a85da844d34be805843bc60d350
I20250625 13:52:03.224027 851 leader_election.cc:304] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:03.224340 1748 raft_consensus.cc:2747] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:03.224627 851 consensus_peers.cc:487] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93f449c80b604a6498f6fd1d1ba4770a. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:03.227689 1029 tablet_service.cc:1468] Processing CreateTablet for tablet d2f545a5934748b2af767d37ea937fca (DEFAULT_TABLE table=default.Table_5f921e5e270c4b868cc0c36c900919d2 [id=612d50114aac44d1ae69ac99b3c0dab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.228132 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2f545a5934748b2af767d37ea937fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.231794 1168 ts_tablet_manager.cc:1918] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:03.231984 1168 log.cc:1199] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/bd5ba76c130a4711af5b21e9eb8562e7
I20250625 13:52:03.233289 1168 ts_tablet_manager.cc:1939] T bd5ba76c130a4711af5b21e9eb8562e7 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:03.234108 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 257) took 6904 ms. Trace:
I20250625 13:52:03.232857 1171 tablet_bootstrap.cc:492] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.234679 1171 ts_tablet_manager.cc:1397] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.280s user 0.003s sys 0.009s
I20250625 13:52:03.235563 1171 raft_consensus.cc:357] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.236292 1171 raft_consensus.cc:383] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.236430 1171 raft_consensus.cc:738] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.236755 1171 consensus_queue.cc:260] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.234352 1168 rpcz_store.cc:276] 0625 13:51:56.329680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.329727 (+ 47us) service_pool.cc:225] Handling call
0625 13:52:03.059885 (+6730158us) ts_tablet_manager.cc:1145] Deleting tablet bd5ba76c130a4711af5b21e9eb8562e7
0625 13:52:03.059900 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.178913 (+119013us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.202899 (+ 23986us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.234073 (+ 31174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6705321}
I20250625 13:52:03.237708 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 256) took 6908 ms. Trace:
I20250625 13:52:03.237682 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet bd5ba76c130a4711af5b21e9eb8562e7 (table default.Table_01ec4a7ff073444b929c6c217632a150 [id=e65eea160d8d473096f157b05be1c294]) successfully deleted
I20250625 13:52:03.237898 1168 rpcz_store.cc:276] 0625 13:51:56.329435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.331360 (+ 1925us) service_pool.cc:225] Handling call
0625 13:52:03.237573 (+6906213us) ts_tablet_manager.cc:1145] Deleting tablet 36b151e200e1430c99ac194bc55881c1
0625 13:52:03.237582 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.237680 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6882748}
W20250625 13:52:03.238332 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 36b151e200e1430c99ac194bc55881c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36b151e200e1430c99ac194bc55881c1 already in progress: creating tablet
I20250625 13:52:03.238410 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 260) took 6580 ms. Trace:
I20250625 13:52:03.238515 1168 rpcz_store.cc:276] 0625 13:51:56.658106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.658224 (+ 118us) service_pool.cc:225] Handling call
0625 13:52:03.238300 (+6580076us) ts_tablet_manager.cc:1145] Deleting tablet 01ba67b500b646d7aa55315f7ee23ed6
0625 13:52:03.238308 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.238386 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6542146}
W20250625 13:52:03.238843 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 01ba67b500b646d7aa55315f7ee23ed6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01ba67b500b646d7aa55315f7ee23ed6 already in progress: creating tablet
I20250625 13:52:03.246212 1016 tablet_service.cc:1468] Processing CreateTablet for tablet bd4cc91d274a45a393beffaaa59a35f0 (DEFAULT_TABLE table=default.Table_88d5d2614816414f8a45b3526e1608d8 [id=677c08587cdd48119667df6818cc2dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.246752 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4cc91d274a45a393beffaaa59a35f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.249143 739 tablet_service.cc:1515] Processing DeleteTablet for tablet f00e72e103f443e886f0df690cff4b6f with delete_type TABLET_DATA_DELETED (replaced by tablet a0e81d4c904248eba858462bcf31c5d9) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.257285 1168 tablet_replica.cc:331] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:03.257440 1171 ts_tablet_manager.cc:1428] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250625 13:52:03.260074 1168 raft_consensus.cc:2241] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.260439 1171 tablet_bootstrap.cc:492] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:03.261758 1168 raft_consensus.cc:2270] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.263083 1168 ts_tablet_manager.cc:1905] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.276166 1171 tablet_bootstrap.cc:654] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.279691 1171 tablet_bootstrap.cc:492] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.284767 1171 ts_tablet_manager.cc:1397] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20250625 13:52:03.285928 1171 raft_consensus.cc:357] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.287132 1171 raft_consensus.cc:383] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.287412 1171 raft_consensus.cc:738] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.291503 1171 consensus_queue.cc:260] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.294565 1171 ts_tablet_manager.cc:1428] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 13:52:03.295233 1171 tablet_bootstrap.cc:492] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:03.297742 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_221603e7750d443aad1f5583b695b1c0" } modify_external_catalogs: true
I20250625 13:52:03.297976 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_221603e7750d443aad1f5583b695b1c0" } modify_external_catalogs: true
I20250625 13:52:03.298547 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9fda2b05f42e471fb687a7c46532b3e4" } modify_external_catalogs: true
I20250625 13:52:03.298728 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9fda2b05f42e471fb687a7c46532b3e4" } modify_external_catalogs: true
I20250625 13:52:03.299225 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0089084e327b47a1a45bf11be8539647" } modify_external_catalogs: true
I20250625 13:52:03.299404 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0089084e327b47a1a45bf11be8539647" } modify_external_catalogs: true
I20250625 13:52:03.248278 1023 tablet_service.cc:1468] Processing CreateTablet for tablet e438e46874f841d2ac972b953ddb4608 (DEFAULT_TABLE table=default.Table_0ca3566f743046db9a91f2a4040f2e03 [id=09abf01183f64858a59a83bf152b107b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.304476 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e438e46874f841d2ac972b953ddb4608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.200562 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 68831b9ad1c14f5e9c954cbd06094896 (DEFAULT_TABLE table=default.Table_4dc8c488c3e44f3f882049d1c0f8feb2 [id=0532faedda28481f9742008aecb6bfd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.307710 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68831b9ad1c14f5e9c954cbd06094896. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.308362 1171 tablet_bootstrap.cc:654] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.308826 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_58c91d644ba94ed580ae053357892c7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.309321 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58c91d644ba94ed580ae053357892c7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.312708 1168 ts_tablet_manager.cc:1918] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.313903 1168 log.cc:1199] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/643928ca4fc94ce58431a0525f2abb99
I20250625 13:52:03.314389 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_09dd798f73ca46cf9edc4fd651169efd" } modify_external_catalogs: true
I20250625 13:52:03.314620 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_09dd798f73ca46cf9edc4fd651169efd" } modify_external_catalogs: true
I20250625 13:52:03.317714 1165 ts_tablet_manager.cc:1918] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.317869 1165 log.cc:1199] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/19017cc372af43b2b5074a8990375ede
I20250625 13:52:03.318224 1165 ts_tablet_manager.cc:1939] T 19017cc372af43b2b5074a8990375ede P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:03.319017 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 529) took 1320 ms. Trace:
I20250625 13:52:03.319120 1165 rpcz_store.cc:276] 0625 13:52:01.998963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.999032 (+ 69us) service_pool.cc:225] Handling call
0625 13:52:02.880977 (+881945us) ts_tablet_manager.cc:1145] Deleting tablet 19017cc372af43b2b5074a8990375ede
0625 13:52:02.880986 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.255082 (+374096us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.317598 (+ 62516us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.318959 (+ 1361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":881651}
I20250625 13:52:03.319681 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 19017cc372af43b2b5074a8990375ede (table default.Table_d675ca14ac884a35aab70e66cdcef846 [id=fed9fb5422bc435ab14ed8f178cc95a1]) successfully deleted
I20250625 13:52:03.324720 1009 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 117) took 1099 ms. Trace:
I20250625 13:52:03.325011 1009 rpcz_store.cc:276] 0625 13:52:02.224782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.224842 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:03.324648 (+1099806us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1619712}
I20250625 13:52:03.327615 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 539) took 1254 ms. Trace:
I20250625 13:52:03.327740 1165 rpcz_store.cc:276] 0625 13:52:02.073123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.074975 (+ 1852us) service_pool.cc:225] Handling call
0625 13:52:03.327395 (+1252420us) ts_tablet_manager.cc:1145] Deleting tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57
0625 13:52:03.327407 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.327551 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252199}
W20250625 13:52:03.328277 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 already in progress: creating tablet
W20250625 13:52:03.329957 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 027b1281543b4d559461e8d222080c30 has no consensus state
W20250625 13:52:03.330576 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 723246e1e6824faf9f9d39c3ea941fe4 consensus state has no leader
I20250625 13:52:03.330840 589 master_service.cc:948] ReplaceTablet: received request to replace tablet ecb9a98e402b4da8a9c2bd8317cef532 from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:03.331223 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af934858a4af4621bb55eb5a528fa14c consensus state has no leader
W20250625 13:52:03.331806 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aef43c4962c747dbb43039d1e706bbd9 consensus state has no leader
W20250625 13:52:03.336234 1048 thread.cc:640] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.041s sys 0.076s
I20250625 13:52:03.340251 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 538) took 1291 ms. Trace:
I20250625 13:52:03.340369 1165 rpcz_store.cc:276] 0625 13:52:02.040901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.041485 (+ 584us) service_pool.cc:225] Handling call
0625 13:52:03.332216 (+1290731us) ts_tablet_manager.cc:1145] Deleting tablet 553af7c7326240838f37e3861d4e3357
0625 13:52:03.332226 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.332327 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255626}
W20250625 13:52:03.340907 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 553af7c7326240838f37e3861d4e3357 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 553af7c7326240838f37e3861d4e3357 already in progress: creating tablet
I20250625 13:52:03.341372 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 537) took 1307 ms. Trace:
I20250625 13:52:03.341467 1165 rpcz_store.cc:276] 0625 13:52:02.033890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.033971 (+ 81us) service_pool.cc:225] Handling call
0625 13:52:03.341199 (+1307228us) ts_tablet_manager.cc:1145] Deleting tablet 48645c913fc24c03a922cf6ea7531022
0625 13:52:03.341210 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.341312 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48,"tablet-delete.queue_time_us":1264604}
W20250625 13:52:03.341913 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 48645c913fc24c03a922cf6ea7531022 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48645c913fc24c03a922cf6ea7531022 already in progress: creating tablet
I20250625 13:52:03.342275 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 540) took 1232 ms. Trace:
I20250625 13:52:03.342373 1165 rpcz_store.cc:276] 0625 13:52:02.109927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.110320 (+ 393us) service_pool.cc:225] Handling call
0625 13:52:03.342134 (+1231814us) ts_tablet_manager.cc:1145] Deleting tablet 9c1deb52366c404cbdc08076d4bcb3d1
0625 13:52:03.342143 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.342223 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229664}
W20250625 13:52:03.342911 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9c1deb52366c404cbdc08076d4bcb3d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c1deb52366c404cbdc08076d4bcb3d1 already in progress: creating tablet
I20250625 13:52:03.343187 1165 tablet_replica.cc:331] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:03.343413 1165 raft_consensus.cc:2241] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.343613 1165 raft_consensus.cc:2270] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.344131 1165 ts_tablet_manager.cc:1905] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.344210 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet d2f545a5934748b2af767d37ea937fca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.349733 1166 tablet_bootstrap.cc:492] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:03.352156 1166 ts_tablet_manager.cc:1397] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.394s user 0.005s sys 0.013s
I20250625 13:52:03.353994 1166 raft_consensus.cc:357] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.277063 1024 tablet_service.cc:1468] Processing CreateTablet for tablet bf7463962f39468c85fb3dc2edcf608f (DEFAULT_TABLE table=default.Table_45ffe0341a354703ab5ab6c66bd27f18 [id=772bd28ed2964d3b9d09189c07cc34f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.285393 1710 ts_tablet_manager.cc:1918] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.287325 1027 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42238 (request call id 560) took 1002 ms. Trace:
I20250625 13:52:03.291932 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 5d478ac74f4f46c6a29a6ac031152498 (DEFAULT_TABLE table=default.Table_a53e9e99051944a1bc66244b53112731 [id=b650eb8998664ad593aba664d70d1898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.293725 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 (DEFAULT_TABLE table=default.Table_d8fbb4df3d0745b6a9671d0dbfb1c789 [id=57a1163f204646f6acf00157f75e5209]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.344424 1170 tablet_bootstrap.cc:492] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:03.344893 983 thread.cc:640] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.048s sys 0.100s
I20250625 13:52:03.263835 1014 tablet_service.cc:1468] Processing CreateTablet for tablet c431abffd05c4136b7510c60eee2a8b1 (DEFAULT_TABLE table=default.Table_d18abaf5b02340fa82d3fee69699e1e3 [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.357427 1166 raft_consensus.cc:383] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.357609 1166 raft_consensus.cc:738] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.358047 1166 consensus_queue.cc:260] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.317052 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 04479dfb34584f0891c604ef72bf93b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.360751 1168 ts_tablet_manager.cc:1939] T 643928ca4fc94ce58431a0525f2abb99 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:03.363204 1166 ts_tablet_manager.cc:1428] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
W20250625 13:52:03.348762 1048 thread.cc:606] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.739s user 0.041s sys 0.076s
I20250625 13:52:03.367637 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 273) took 6673 ms. Trace:
I20250625 13:52:03.369778 1027 rpcz_store.cc:276] 0625 13:52:02.284920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.396986 (+112066us) service_pool.cc:225] Handling call
0625 13:52:02.672958 (+275972us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0625 13:52:02.672984 (+ 26us) ts_tablet_manager.cc:676] Creating new metadata...
0625 13:52:02.947535 (+274551us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.287060 (+339525us) tablet_replica.cc:206] Creating consensus instance
0625 13:52:03.287244 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:52:03.369733 1168 rpcz_store.cc:276] 0625 13:51:56.693889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.765106 (+ 71217us) service_pool.cc:225] Handling call
0625 13:52:03.257201 (+6492095us) ts_tablet_manager.cc:1145] Deleting tablet 643928ca4fc94ce58431a0525f2abb99
0625 13:52:03.257214 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.281544 (+ 24330us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.312597 (+ 31053us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.367583 (+ 54986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6405141}
I20250625 13:52:03.370311 1710 log.cc:1199] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/f73f420a4d7a438eadde3d6af64f0a11
I20250625 13:52:03.370342 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 643928ca4fc94ce58431a0525f2abb99 (table default.Table_ad4e07371e3b42c5b05954f9537fd6b7 [id=c861ee66493e47ec97ad3aa9a6495499]) successfully deleted
I20250625 13:52:03.370967 1170 ts_tablet_manager.cc:1397] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.591s user 0.006s sys 0.003s
W20250625 13:52:03.372426 983 thread.cc:606] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.691s user 0.048s sys 0.100s
I20250625 13:52:03.372694 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e03511e0ab134ad799672ced1a244df2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.372957 1166 tablet_bootstrap.cc:492] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:03.373066 1518 raft_consensus.cc:491] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.373232 1518 raft_consensus.cc:513] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:03.373370 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e03511e0ab134ad799672ced1a244df2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.373847 1518 leader_election.cc:290] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:03.374323 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1535cba3bd964e11bf3c6198b8d6f785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.374871 1710 ts_tablet_manager.cc:1939] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:03.375097 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b9f454bddd8a463b863eb0ca65ebf3af" } modify_external_catalogs: true
I20250625 13:52:03.375200 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 518) took 1608 ms. Trace:
I20250625 13:52:03.375293 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b9f454bddd8a463b863eb0ca65ebf3af" } modify_external_catalogs: true
I20250625 13:52:03.375327 1710 rpcz_store.cc:276] 0625 13:52:01.766322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.766389 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:02.694001 (+927612us) ts_tablet_manager.cc:1145] Deleting tablet f73f420a4d7a438eadde3d6af64f0a11
0625 13:52:02.694008 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:02.797272 (+103264us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x616000033d54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf7f000 00007f18fd291093 00007f18ffd6e137 00007f18ffda01c4 00007f18ffd81c20 00007f18ffd819ad 0000000000a7573c 00007f18f05649df 00007f18f0567cab 00007f18f0567b28
0625 13:52:03.103816 (+306544us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.285262 (+181446us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.375167 (+ 89905us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95616768,"tablet-delete.queue_time_us":927396}
I20250625 13:52:03.375838 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet f73f420a4d7a438eadde3d6af64f0a11 (table default.Table_9e86330e77034242a9aa4fbd274c3151 [id=0b06b619484e459f949d8acb2bb19512]) successfully deleted
I20250625 13:52:03.376107 1166 tablet_bootstrap.cc:654] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.376099 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7c55c56331834053a65654bb59304b61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.376528 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c55c56331834053a65654bb59304b61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.376382 1710 tablet_replica.cc:331] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:03.376945 985 leader_election.cc:343] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 04479dfb34584f0891c604ef72bf93b5
I20250625 13:52:03.377179 1710 raft_consensus.cc:2241] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:03.377713 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1535cba3bd964e11bf3c6198b8d6f785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.378180 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 27092991fcad475fabc77d1f8bab0582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.379105 1710 raft_consensus.cc:2270] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.379770 1710 ts_tablet_manager.cc:1905] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.380095 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7463962f39468c85fb3dc2edcf608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.382172 1165 ts_tablet_manager.cc:1918] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.382318 1165 log.cc:1199] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/6771d8e180fe48569a0e016db0cc9b51
I20250625 13:52:03.382707 1165 ts_tablet_manager.cc:1939] T 6771d8e180fe48569a0e016db0cc9b51 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:03.382719 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ae4a88bee59a4776b7bf5b7fb6b31721"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.383203 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae4a88bee59a4776b7bf5b7fb6b31721 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.383473 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 544) took 1253 ms. Trace:
I20250625 13:52:03.383606 1165 rpcz_store.cc:276] 0625 13:52:02.129804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.130915 (+ 1111us) service_pool.cc:225] Handling call
0625 13:52:03.343110 (+1212195us) ts_tablet_manager.cc:1145] Deleting tablet 6771d8e180fe48569a0e016db0cc9b51
0625 13:52:03.343118 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.378891 (+ 35773us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.382039 (+ 3148us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.383437 (+ 1398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207441}
I20250625 13:52:03.384022 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 549) took 1230 ms. Trace:
I20250625 13:52:03.384152 1165 rpcz_store.cc:276] 0625 13:52:02.154019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.183634 (+ 29615us) service_pool.cc:225] Handling call
0625 13:52:03.383875 (+1200241us) ts_tablet_manager.cc:1145] Deleting tablet fb4973cbffc743dba98d7b7c9f76be3f
0625 13:52:03.383884 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.383981 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175793}
I20250625 13:52:03.384476 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d478ac74f4f46c6a29a6ac031152498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.384521 1776 raft_consensus.cc:491] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.384680 1776 raft_consensus.cc:513] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.385375 1776 leader_election.cc:290] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:03.386068 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01ba67b500b646d7aa55315f7ee23ed6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:03.386312 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]). Expected version 1 got 0
W20250625 13:52:03.386574 711 leader_election.cc:343] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.387224 1027 tablet_service.cc:1468] Processing CreateTablet for tablet d7660a2b70744bd589220b28e6edbc42 (DEFAULT_TABLE table=default.Table_ebff453083b745618b11918f62df6a0a [id=e47d3a1372d84468b1145fecbb6d37b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.387621 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7660a2b70744bd589220b28e6edbc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.389161 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c431abffd05c4136b7510c60eee2a8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.391829 757 tablet_service.cc:1468] Processing CreateTablet for tablet d7660a2b70744bd589220b28e6edbc42 (DEFAULT_TABLE table=default.Table_ebff453083b745618b11918f62df6a0a [id=e47d3a1372d84468b1145fecbb6d37b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.392001 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 6771d8e180fe48569a0e016db0cc9b51 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]) successfully deleted
I20250625 13:52:03.392225 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7660a2b70744bd589220b28e6edbc42. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:03.392287 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet fb4973cbffc743dba98d7b7c9f76be3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb4973cbffc743dba98d7b7c9f76be3f already in progress: creating tablet
I20250625 13:52:03.393888 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_70ec24bdd5764fa1893a92a1e341a2ea" } new_table_name: "default.Table_fb12128ffae94ceabd5af53e908db5d2" modify_external_catalogs: true
I20250625 13:52:03.397004 746 tablet_service.cc:1468] Processing CreateTablet for tablet f1d647c2504c494d8b60c751126ee825 (DEFAULT_TABLE table=default.Table_003becbbf12b448b8d9581a65994b657 [id=360865a660714d4cbc3d58aa56de7051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.397388 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d647c2504c494d8b60c751126ee825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.400959 743 tablet_service.cc:1468] Processing CreateTablet for tablet 23a2b56c117f49d3b7bab4bb3d776173 (DEFAULT_TABLE table=default.Table_f13b94122bf947aab60dbf0eb035cc63 [id=c41c875b733941ba83261e1e9853583c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.401382 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a2b56c117f49d3b7bab4bb3d776173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.402148 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3016763b9e294b9da2861fe97e57e241
I20250625 13:52:03.402910 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_583f49dbf61c498b98a3cd236437318d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.403522 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_583f49dbf61c498b98a3cd236437318d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.403618 747 tablet_service.cc:1468] Processing CreateTablet for tablet 687fcd13a80c466983d544c0453f6f7d (DEFAULT_TABLE table=default.Table_1caed92374d04bc3a365d46eb965441a [id=703bddf2071145d3abb1f52b16213d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.403993 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687fcd13a80c466983d544c0453f6f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.404774 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 93fdde7d068b400ba7070166eff63502
I20250625 13:52:03.405467 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_74dfe85879a04d7eb5bee2e8f00065ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.405944 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74dfe85879a04d7eb5bee2e8f00065ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.407605 756 tablet_service.cc:1468] Processing CreateTablet for tablet c0f461a7a7ef45009279716a3024ee26 (DEFAULT_TABLE table=default.Table_9fda2b05f42e471fb687a7c46532b3e4 [id=de3f38f08f954fd1afbfeb9a20118146]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.408015 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f461a7a7ef45009279716a3024ee26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.412073 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 23a2b56c117f49d3b7bab4bb3d776173 (DEFAULT_TABLE table=default.Table_f13b94122bf947aab60dbf0eb035cc63 [id=c41c875b733941ba83261e1e9853583c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.412736 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f197ea67c34546a7981bcd9df94702e6
I20250625 13:52:03.413261 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 6771d8e180fe48569a0e016db0cc9b51 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:03.413486 744 tablet_service.cc:1468] Processing CreateTablet for tablet 5c4458791d4a4f0690fd9a4f31b896f3 (DEFAULT_TABLE table=default.Table_c8d20a37f388454db75fc0c021da9321 [id=fb8a0d9552d142d091c2875a5e4f2a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.413856 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c4458791d4a4f0690fd9a4f31b896f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.413849 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c0f461a7a7ef45009279716a3024ee26
I20250625 13:52:03.414986 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0e77736191544e8da32ca9927f0a5804
I20250625 13:52:03.419354 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.419595 740 tablet_service.cc:1468] Processing CreateTablet for tablet 317707085185470c9b096892d8e5d67f (DEFAULT_TABLE table=default.Table_ce91dd339b4c4261bdc2fe9e91b0d413 [id=9616603f14ce4444b558b5495c792bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.419921 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 317707085185470c9b096892d8e5d67f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.423085 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 5c4458791d4a4f0690fd9a4f31b896f3 (DEFAULT_TABLE table=default.Table_c8d20a37f388454db75fc0c021da9321 [id=fb8a0d9552d142d091c2875a5e4f2a74]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:03.423648 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22ee97e6804a4b9fb564ed6f226bc1e0 consensus state has no leader
I20250625 13:52:03.423607 745 tablet_service.cc:1468] Processing CreateTablet for tablet 6c331c57bff44091be2849c21d6ab2d7 (DEFAULT_TABLE table=default.Table_f37637a5cbf94eb986a689ea87854c41 [id=a6c88962a9ea4c9cb6f03bec937372ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.423964 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c331c57bff44091be2849c21d6ab2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.427242 1170 raft_consensus.cc:357] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.427428 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3016763b9e294b9da2861fe97e57e241 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.427659 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 93fdde7d068b400ba7070166eff63502 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.428035 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9def3e1e149a4045a1bb2da76ced8eaf (DEFAULT_TABLE table=default.Table_82770759b43b46a0850155827b18c124 [id=d46ae4e60d52433783b3934a1f48b3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.428366 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9def3e1e149a4045a1bb2da76ced8eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.428318 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 3016763b9e294b9da2861fe97e57e241 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.428728 1020 tablet_service.cc:1468] Processing CreateTablet for tablet f1d647c2504c494d8b60c751126ee825 (DEFAULT_TABLE table=default.Table_003becbbf12b448b8d9581a65994b657 [id=360865a660714d4cbc3d58aa56de7051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.431172 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 317707085185470c9b096892d8e5d67f (DEFAULT_TABLE table=default.Table_ce91dd339b4c4261bdc2fe9e91b0d413 [id=9616603f14ce4444b558b5495c792bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.431396 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 6c331c57bff44091be2849c21d6ab2d7 (DEFAULT_TABLE table=default.Table_f37637a5cbf94eb986a689ea87854c41 [id=a6c88962a9ea4c9cb6f03bec937372ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.431593 741 tablet_service.cc:1468] Processing CreateTablet for tablet 3dfb86bb8bdc4e4d80cee8e155d63353 (DEFAULT_TABLE table=default.Table_809755ea1ee14781acbef7ee9ccf5548 [id=d1ec6df4eee74adca3cca8eba018e258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.431937 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dfb86bb8bdc4e4d80cee8e155d63353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.433705 1776 raft_consensus.cc:491] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.433887 1776 raft_consensus.cc:513] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.434506 1776 leader_election.cc:290] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:03.436316 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 93fdde7d068b400ba7070166eff63502 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.436797 1170 raft_consensus.cc:383] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.439405 752 tablet_service.cc:1468] Processing CreateTablet for tablet 81a359c798024b2a99aa3b307cc1967f (DEFAULT_TABLE table=default.Table_e170ed64e7174c39a3e21ceac0e51021 [id=ebbdfe16e6f64ccd95e6dc7d068cd814]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.439793 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a359c798024b2a99aa3b307cc1967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.436997 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 687fcd13a80c466983d544c0453f6f7d (DEFAULT_TABLE table=default.Table_1caed92374d04bc3a365d46eb965441a [id=703bddf2071145d3abb1f52b16213d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.437696 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 9def3e1e149a4045a1bb2da76ced8eaf (DEFAULT_TABLE table=default.Table_82770759b43b46a0850155827b18c124 [id=d46ae4e60d52433783b3934a1f48b3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.439227 1011 tablet_service.cc:1468] Processing CreateTablet for tablet bee9cd33bff04a69ac66ba23d9577603 (DEFAULT_TABLE table=default.Table_b1982b3d458a4586a464994f13dd1481 [id=4055346986414eec9af02674c2dc1628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.443884 739 tablet_service.cc:1468] Processing CreateTablet for tablet 027b1281543b4d559461e8d222080c30 (DEFAULT_TABLE table=default.Table_49a5b264518840179315082ba4922b6c [id=dec0d3288d724ffeb90336093522ce51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.444092 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 81a359c798024b2a99aa3b307cc1967f (DEFAULT_TABLE table=default.Table_e170ed64e7174c39a3e21ceac0e51021 [id=ebbdfe16e6f64ccd95e6dc7d068cd814]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.444411 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027b1281543b4d559461e8d222080c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.445853 1170 raft_consensus.cc:738] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.446175 1170 consensus_queue.cc:260] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.446846 750 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f461a7a7ef45009279716a3024ee26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.447124 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e77736191544e8da32ca9927f0a5804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.447927 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a2b56c117f49d3b7bab4bb3d776173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.445235 1018 tablet_service.cc:1468] Processing CreateTablet for tablet c0f461a7a7ef45009279716a3024ee26 (DEFAULT_TABLE table=default.Table_9fda2b05f42e471fb687a7c46532b3e4 [id=de3f38f08f954fd1afbfeb9a20118146]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.450909 1776 raft_consensus.cc:491] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.451061 1776 raft_consensus.cc:513] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.451742 1776 leader_election.cc:290] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:03.451784 1170 ts_tablet_manager.cc:1428] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.076s user 0.003s sys 0.000s
I20250625 13:52:03.451833 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3dfb86bb8bdc4e4d80cee8e155d63353 (DEFAULT_TABLE table=default.Table_809755ea1ee14781acbef7ee9ccf5548 [id=d1ec6df4eee74adca3cca8eba018e258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.452112 1170 tablet_bootstrap.cc:492] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:03.452178 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 027b1281543b4d559461e8d222080c30 (DEFAULT_TABLE table=default.Table_49a5b264518840179315082ba4922b6c [id=dec0d3288d724ffeb90336093522ce51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.455178 877 tablet_service.cc:1468] Processing CreateTablet for tablet f1d647c2504c494d8b60c751126ee825 (DEFAULT_TABLE table=default.Table_003becbbf12b448b8d9581a65994b657 [id=360865a660714d4cbc3d58aa56de7051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.456346 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01ba67b500b646d7aa55315f7ee23ed6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:03.456681 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf1664afbab647b3baeed6391196bed9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:03.457567 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c5bea3b6214fa8af2792c5e9c411d9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:03.462460 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d647c2504c494d8b60c751126ee825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.462711 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee9cd33bff04a69ac66ba23d9577603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.462908 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dfb86bb8bdc4e4d80cee8e155d63353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.463471 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027b1281543b4d559461e8d222080c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.464129 891 tablet_service.cc:1468] Processing CreateTablet for tablet 81a359c798024b2a99aa3b307cc1967f (DEFAULT_TABLE table=default.Table_e170ed64e7174c39a3e21ceac0e51021 [id=ebbdfe16e6f64ccd95e6dc7d068cd814]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.464468 893 tablet_service.cc:1468] Processing CreateTablet for tablet d7660a2b70744bd589220b28e6edbc42 (DEFAULT_TABLE table=default.Table_ebff453083b745618b11918f62df6a0a [id=e47d3a1372d84468b1145fecbb6d37b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.464640 896 tablet_service.cc:1468] Processing CreateTablet for tablet bee9cd33bff04a69ac66ba23d9577603 (DEFAULT_TABLE table=default.Table_b1982b3d458a4586a464994f13dd1481 [id=4055346986414eec9af02674c2dc1628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.465718 879 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f461a7a7ef45009279716a3024ee26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.466146 881 tablet_service.cc:1515] Processing DeleteTablet for tablet f197ea67c34546a7981bcd9df94702e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.466291 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f033bfa4925b4d1c971789eb77efd92c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.466383 894 tablet_service.cc:1468] Processing CreateTablet for tablet 5c4458791d4a4f0690fd9a4f31b896f3 (DEFAULT_TABLE table=default.Table_c8d20a37f388454db75fc0c021da9321 [id=fb8a0d9552d142d091c2875a5e4f2a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.467515 889 tablet_service.cc:1468] Processing CreateTablet for tablet 23a2b56c117f49d3b7bab4bb3d776173 (DEFAULT_TABLE table=default.Table_f13b94122bf947aab60dbf0eb035cc63 [id=c41c875b733941ba83261e1e9853583c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.467794 880 tablet_service.cc:1468] Processing CreateTablet for tablet c0f461a7a7ef45009279716a3024ee26 (DEFAULT_TABLE table=default.Table_9fda2b05f42e471fb687a7c46532b3e4 [id=de3f38f08f954fd1afbfeb9a20118146]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.468010 887 tablet_service.cc:1468] Processing CreateTablet for tablet 027b1281543b4d559461e8d222080c30 (DEFAULT_TABLE table=default.Table_49a5b264518840179315082ba4922b6c [id=dec0d3288d724ffeb90336093522ce51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.469135 892 tablet_service.cc:1468] Processing CreateTablet for tablet 9def3e1e149a4045a1bb2da76ced8eaf (DEFAULT_TABLE table=default.Table_82770759b43b46a0850155827b18c124 [id=d46ae4e60d52433783b3934a1f48b3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.468338 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f73f420a4d7a438eadde3d6af64f0a11" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:03.469990 884 tablet_service.cc:1468] Processing CreateTablet for tablet 317707085185470c9b096892d8e5d67f (DEFAULT_TABLE table=default.Table_ce91dd339b4c4261bdc2fe9e91b0d413 [id=9616603f14ce4444b558b5495c792bb4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:03.470960 711 leader_election.cc:343] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.470963 888 tablet_service.cc:1468] Processing CreateTablet for tablet 687fcd13a80c466983d544c0453f6f7d (DEFAULT_TABLE table=default.Table_1caed92374d04bc3a365d46eb965441a [id=703bddf2071145d3abb1f52b16213d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.471112 711 leader_election.cc:304] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:03.471592 886 tablet_service.cc:1468] Processing CreateTablet for tablet 6c331c57bff44091be2849c21d6ab2d7 (DEFAULT_TABLE table=default.Table_f37637a5cbf94eb986a689ea87854c41 [id=a6c88962a9ea4c9cb6f03bec937372ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.472340 881 tablet_service.cc:1468] Processing CreateTablet for tablet 3dfb86bb8bdc4e4d80cee8e155d63353 (DEFAULT_TABLE table=default.Table_809755ea1ee14781acbef7ee9ccf5548 [id=d1ec6df4eee74adca3cca8eba018e258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.472618 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e77736191544e8da32ca9927f0a5804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.472103 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 261) took 6681 ms. Trace:
I20250625 13:52:03.473089 1776 raft_consensus.cc:2747] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.473572 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d647c2504c494d8b60c751126ee825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.473644 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f00e72e103f443e886f0df690cff4b6f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:03.473989 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee9cd33bff04a69ac66ba23d9577603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.474351 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 3016763b9e294b9da2861fe97e57e241 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.475260 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 317707085185470c9b096892d8e5d67f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:03.475243 711 leader_election.cc:343] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.475126 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27092991fcad475fabc77d1f8bab0582" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:03.475679 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04479dfb34584f0891c604ef72bf93b5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:03.475854 913 raft_consensus.cc:2466] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:03.476337 912 raft_consensus.cc:3058] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:03.476379 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 93fdde7d068b400ba7070166eff63502 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.476565 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c4458791d4a4f0690fd9a4f31b896f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.476684 983 leader_election.cc:304] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:03.477432 711 leader_election.cc:343] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.479457 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c331c57bff44091be2849c21d6ab2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.479458 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27092991fcad475fabc77d1f8bab0582" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:03.479933 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f461a7a7ef45009279716a3024ee26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.479943 983 leader_election.cc:304] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:03.480099 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet f197ea67c34546a7981bcd9df94702e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.480342 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f00e72e103f443e886f0df690cff4b6f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:03.481307 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23a2b56c117f49d3b7bab4bb3d776173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.481877 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f461a7a7ef45009279716a3024ee26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.482115 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e77736191544e8da32ca9927f0a5804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.482613 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 04479dfb34584f0891c604ef72bf93b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.483529 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687fcd13a80c466983d544c0453f6f7d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:03.483621 711 leader_election.cc:343] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.483819 711 leader_election.cc:304] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:03.484455 1776 raft_consensus.cc:2747] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.486763 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dfb86bb8bdc4e4d80cee8e155d63353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.486981 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9def3e1e149a4045a1bb2da76ced8eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.487444 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027b1281543b4d559461e8d222080c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.487856 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 317707085185470c9b096892d8e5d67f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.488263 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7660a2b70744bd589220b28e6edbc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.488639 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a359c798024b2a99aa3b307cc1967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.493161 1518 raft_consensus.cc:2747] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.493467 1518 raft_consensus.cc:2747] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.494153 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c331c57bff44091be2849c21d6ab2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.495702 741 tablet_service.cc:1515] Processing DeleteTablet for tablet f197ea67c34546a7981bcd9df94702e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:03.498281 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 430c3bed360b4f32855a8adb99076608 (table default.Table_1099b7a2614540d3ac476e1bad30554c [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 430c3bed360b4f32855a8adb99076608 consensus state has no leader
I20250625 13:52:03.503058 757 tablet_service.cc:1468] Processing CreateTablet for tablet bee9cd33bff04a69ac66ba23d9577603 (DEFAULT_TABLE table=default.Table_b1982b3d458a4586a464994f13dd1481 [id=4055346986414eec9af02674c2dc1628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.503521 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee9cd33bff04a69ac66ba23d9577603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.510151 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d67dfdd23e284e7588f0d5fade2e36e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.510870 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67dfdd23e284e7588f0d5fade2e36e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:03.512233 711 leader_election.cc:343] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.512135 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bd76ad95ad844003ad3afa39db18c7f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.512398 711 leader_election.cc:304] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:03.512689 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd76ad95ad844003ad3afa39db18c7f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.512948 1776 raft_consensus.cc:2747] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.513432 912 raft_consensus.cc:2466] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:03.515511 1165 tablet_replica.cc:331] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:03.515743 1165 raft_consensus.cc:2241] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.515981 1165 raft_consensus.cc:2270] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.516587 1165 ts_tablet_manager.cc:1905] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.523837 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4190488f5e9f436d986974482c2e5ecd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.527428 1748 consensus_queue.cc:1035] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:03.528486 1166 tablet_bootstrap.cc:492] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:03.528736 1166 ts_tablet_manager.cc:1397] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.156s user 0.004s sys 0.000s
I20250625 13:52:03.529664 1166 raft_consensus.cc:357] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:03.530098 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4190488f5e9f436d986974482c2e5ecd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.530258 1166 raft_consensus.cc:383] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.530360 1166 raft_consensus.cc:738] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.530623 1166 consensus_queue.cc:260] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.535833 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3034d62d25c74b3a9d694ce4d7ba717c" } new_table_name: "default.Table_67a5cc5e1f4449029550030a01425b01" modify_external_catalogs: true
I20250625 13:52:03.536147 1166 ts_tablet_manager.cc:1428] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:03.537382 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1535cba3bd964e11bf3c6198b8d6f785" } new_table_name: "default.Table_21b38804564840edbbc09582cfe8f506" modify_external_catalogs: true
I20250625 13:52:03.538673 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1653a8a7a93d4e77abf24438181cb8ec" } new_table_name: "default.Table_3642d962b3544fc58d26f3d5592639b9" modify_external_catalogs: true
I20250625 13:52:03.538784 1166 tablet_bootstrap.cc:492] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:03.540827 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9def3e1e149a4045a1bb2da76ced8eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.541313 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cb91eec0b55a4cd1b261ee9f113a4644" } new_table_name: "default.Table_3c5334740bcc46cdb576090f3911074a" modify_external_catalogs: true
W20250625 13:52:03.546206 851 consensus_peers.cc:487] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:03.553692 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 with delete_type TABLET_DATA_DELETED (replaced by tablet c431abffd05c4136b7510c60eee2a8b1) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.553934 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 687fcd13a80c466983d544c0453f6f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.563375 1168 rpcz_store.cc:276] 0625 13:51:56.688776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.691086 (+ 2310us) service_pool.cc:225] Handling call
0625 13:52:03.370090 (+6679004us) ts_tablet_manager.cc:1145] Deleting tablet 991bb428def545fd979e9f1a80ce4d18
0625 13:52:03.370101 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.370236 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6515280}
W20250625 13:52:03.564056 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 991bb428def545fd979e9f1a80ce4d18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 991bb428def545fd979e9f1a80ce4d18 already in progress: creating tablet
I20250625 13:52:03.564581 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 270) took 6874 ms. Trace:
I20250625 13:52:03.564680 1168 rpcz_store.cc:276] 0625 13:51:56.689857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.696191 (+ 6334us) service_pool.cc:225] Handling call
0625 13:52:03.564384 (+6868193us) ts_tablet_manager.cc:1145] Deleting tablet 3b0037d09796455cbaaadfaa9c010673
0625 13:52:03.564400 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.564524 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6708232}
W20250625 13:52:03.565135 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3b0037d09796455cbaaadfaa9c010673 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b0037d09796455cbaaadfaa9c010673 already in progress: creating tablet
I20250625 13:52:03.565457 1168 tablet_replica.cc:331] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:03.565663 1168 raft_consensus.cc:2241] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.565851 1168 raft_consensus.cc:2270] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.566368 1168 ts_tablet_manager.cc:1905] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.569458 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 50786a4c25274ffb8f9910c5dafe3933 (table default.Table_ebb1cd6aa381410a82cac38e7197353c [id=d7cbe338b2554a6e99b07325632aaa33]). Expected version 3 got 0
I20250625 13:52:03.584580 1165 ts_tablet_manager.cc:1918] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250625 13:52:03.585477 1165 log.cc:1199] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/6e0f2fa5d5614ad394e6e12ec62814d4
I20250625 13:52:03.585865 1165 ts_tablet_manager.cc:1939] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:03.586654 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 564) took 1283 ms. Trace:
I20250625 13:52:03.586740 1165 rpcz_store.cc:276] 0625 13:52:02.303623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.304266 (+ 643us) service_pool.cc:225] Handling call
0625 13:52:03.515441 (+1211175us) ts_tablet_manager.cc:1145] Deleting tablet 6e0f2fa5d5614ad394e6e12ec62814d4
0625 13:52:03.515453 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.568986 (+ 53533us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.584443 (+ 15457us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.586632 (+ 2189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210964}
I20250625 13:52:03.587178 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd]) successfully deleted
I20250625 13:52:03.588044 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 565) took 1276 ms. Trace:
I20250625 13:52:03.589811 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_35b3cb8e94364083971a19c3e8bc98a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.590356 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35b3cb8e94364083971a19c3e8bc98a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.594074 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c4458791d4a4f0690fd9a4f31b896f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.595683 1165 rpcz_store.cc:276] 0625 13:52:02.311272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.312481 (+ 1209us) service_pool.cc:225] Handling call
0625 13:52:03.587902 (+1275421us) ts_tablet_manager.cc:1145] Deleting tablet 69672579a83c41d09ee7c78a6fa7d746
0625 13:52:03.587912 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.588025 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275041}
W20250625 13:52:03.596254 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 69672579a83c41d09ee7c78a6fa7d746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69672579a83c41d09ee7c78a6fa7d746 already in progress: creating tablet
I20250625 13:52:03.597295 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 566) took 1251 ms. Trace:
W20250625 13:52:03.598763 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 156da2a965c64d1090a4f9d4eafdec70 has no consensus state
W20250625 13:52:03.599285 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e2711256c9042ebb651ae0ca3d02c89 has no consensus state
I20250625 13:52:03.599483 1165 rpcz_store.cc:276] 0625 13:52:02.345720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.345795 (+ 75us) service_pool.cc:225] Handling call
0625 13:52:03.597155 (+1251360us) ts_tablet_manager.cc:1145] Deleting tablet 8063122ea7b24b6790f5c3cc884e09be
0625 13:52:03.597166 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.597275 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1250930}
W20250625 13:52:03.599967 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8063122ea7b24b6790f5c3cc884e09be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8063122ea7b24b6790f5c3cc884e09be already in progress: creating tablet
I20250625 13:52:03.600411 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 567) took 1238 ms. Trace:
W20250625 13:52:03.601338 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7bd4939714a4359846345ed0521701c consensus state has no leader
I20250625 13:52:03.602298 1165 rpcz_store.cc:276] 0625 13:52:02.361842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.362325 (+ 483us) service_pool.cc:225] Handling call
0625 13:52:03.600277 (+1237952us) ts_tablet_manager.cc:1145] Deleting tablet 5d4fa8abea8d4eee8651be9bf4b2b34f
0625 13:52:03.600288 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.600392 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237456}
W20250625 13:52:03.603375 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d4fa8abea8d4eee8651be9bf4b2b34f already in progress: creating tablet
W20250625 13:52:03.603665 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54cc2d0cf95b41d6b36483af5436fcde consensus state has no leader
I20250625 13:52:03.604103 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 569) took 1152 ms. Trace:
I20250625 13:52:03.604792 1171 tablet_bootstrap.cc:492] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.605020 1171 ts_tablet_manager.cc:1397] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.310s user 0.000s sys 0.007s
I20250625 13:52:03.605085 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a359c798024b2a99aa3b307cc1967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.605357 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f51c5c5fe5394b598404b1944ee7d0b5" } modify_external_catalogs: true
I20250625 13:52:03.605544 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f51c5c5fe5394b598404b1944ee7d0b5" } modify_external_catalogs: true
I20250625 13:52:03.605841 1171 raft_consensus.cc:357] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.606065 1748 raft_consensus.cc:491] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.606246 1748 raft_consensus.cc:513] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.606909 1748 leader_election.cc:290] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:03.606220 1171 raft_consensus.cc:383] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.607576 1171 raft_consensus.cc:738] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.607890 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fb311c42df024421b325596e0cb7a778" } modify_external_catalogs: true
I20250625 13:52:03.607911 1171 consensus_queue.cc:260] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.608060 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fb311c42df024421b325596e0cb7a778" } modify_external_catalogs: true
W20250625 13:52:03.608748 851 leader_election.cc:343] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f033bfa4925b4d1c971789eb77efd92c
I20250625 13:52:03.608098 1165 rpcz_store.cc:276] 0625 13:52:02.451328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.451376 (+ 48us) service_pool.cc:225] Handling call
0625 13:52:03.603966 (+1152590us) ts_tablet_manager.cc:1145] Deleting tablet a6ff3b675aba42adb035ff5d1404a889
0625 13:52:03.603978 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.604082 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152236}
W20250625 13:52:03.611471 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a6ff3b675aba42adb035ff5d1404a889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6ff3b675aba42adb035ff5d1404a889 already in progress: creating tablet
I20250625 13:52:03.611997 1165 tablet_replica.cc:331] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:03.612224 1165 raft_consensus.cc:2241] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.612443 1165 raft_consensus.cc:2270] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.612865 1171 ts_tablet_manager.cc:1428] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:52:03.613256 1171 tablet_bootstrap.cc:492] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:03.615720 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_796954f1442545d2aa7d7ffaf74e29d2" } modify_external_catalogs: true
I20250625 13:52:03.615895 1165 ts_tablet_manager.cc:1905] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.615912 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_796954f1442545d2aa7d7ffaf74e29d2" } modify_external_catalogs: true
I20250625 13:52:03.616814 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6a3e9fc3400f4884a44a9054a64afe1c" } modify_external_catalogs: true
I20250625 13:52:03.617029 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6a3e9fc3400f4884a44a9054a64afe1c" } modify_external_catalogs: true
I20250625 13:52:03.617777 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_349a67a1a0284f84ac46a166fb498b07" } modify_external_catalogs: true
I20250625 13:52:03.617933 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_349a67a1a0284f84ac46a166fb498b07" } modify_external_catalogs: true
I20250625 13:52:03.626658 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e90d64fdf05e442aa9da241d5fa1263b" } new_table_name: "default.Table_9222dfc69aed449eb67eb228923b064f" modify_external_catalogs: true
I20250625 13:52:03.628479 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f461a7a7ef45009279716a3024ee26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.634865 1776 raft_consensus.cc:491] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.635020 1776 raft_consensus.cc:513] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.635766 1776 leader_election.cc:290] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:03.636982 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae4a105e80c1460cbb21e1bdb9b356b1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:03.637579 711 leader_election.cc:343] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.637866 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae4a105e80c1460cbb21e1bdb9b356b1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:03.639358 757 tablet_service.cc:1468] Processing CreateTablet for tablet f14db116c8184370a995e82e33813932 (DEFAULT_TABLE table=default.Table_58c91d644ba94ed580ae053357892c7a [id=5328443f7f5742ba84e8a4b660ca6f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.639468 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cbdacccee68e4114ab2784c7bda953df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.639588 711 leader_election.cc:343] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.639689 711 leader_election.cc:304] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:03.639796 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14db116c8184370a995e82e33813932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.640071 1776 raft_consensus.cc:2747] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:03.640444 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbdacccee68e4114ab2784c7bda953df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.642134 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c798e33a708c4d87b816a43d3a9d2d2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.642185 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_679758c0160345d48f8a3bfc5a1a042d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.642658 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_679758c0160345d48f8a3bfc5a1a042d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:03.642701 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c798e33a708c4d87b816a43d3a9d2d2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.644933 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f14db116c8184370a995e82e33813932 (DEFAULT_TABLE table=default.Table_58c91d644ba94ed580ae053357892c7a [id=5328443f7f5742ba84e8a4b660ca6f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.645393 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14db116c8184370a995e82e33813932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.629617 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f033bfa4925b4d1c971789eb77efd92c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:03.647187 896 tablet_service.cc:1468] Processing CreateTablet for tablet f14db116c8184370a995e82e33813932 (DEFAULT_TABLE table=default.Table_58c91d644ba94ed580ae053357892c7a [id=5328443f7f5742ba84e8a4b660ca6f13]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:03.647408 849 leader_election.cc:343] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.647559 849 leader_election.cc:304] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:03.647723 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f14db116c8184370a995e82e33813932. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.648012 1748 raft_consensus.cc:2747] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.649292 752 tablet_service.cc:1468] Processing CreateTablet for tablet 197e33dc3bad45cb9db7286b472a7111 (DEFAULT_TABLE table=default.Table_af1cd42de11e4022ac9d9550faca59b6 [id=64915e8dad4946e1bd552eba5b285c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.649680 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197e33dc3bad45cb9db7286b472a7111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.651598 743 tablet_service.cc:1468] Processing CreateTablet for tablet c0f8e222d6aa4f5582688f874ce04f4f (DEFAULT_TABLE table=default.Table_7c37f5cca4df4a929018f7a81e5cdcb3 [id=6d7c623de5b041e6b8ac74486e0c514b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.651901 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f8e222d6aa4f5582688f874ce04f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.652137 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_50ec7255c1514670a780c11526d562a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.652666 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50ec7255c1514670a780c11526d562a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.654614 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 5e13a05108d545629f0a5e206dca270c (DEFAULT_TABLE table=default.Table_97642fc925bc492e9be22bec273710f0 [id=f75dad2552f846a297fcc0ea3b622d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.655023 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e13a05108d545629f0a5e206dca270c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.655202 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 4616015b587a459abfb5403b690d23cf (DEFAULT_TABLE table=default.Table_2a0fc7dfe9884d9eb17952b2d9d3d5ee [id=612c57f12aef41cea072c4ae51552545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.655572 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4616015b587a459abfb5403b690d23cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.655560 739 tablet_service.cc:1468] Processing CreateTablet for tablet 5af2b974df3a41f7b60537c9d8eac1e6 (DEFAULT_TABLE table=default.Table_f6b4ad595d094974adb62da4fe761a46 [id=df7316cd20aa497c9ad70964acb1b03f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.655886 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af2b974df3a41f7b60537c9d8eac1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.656180 1027 tablet_service.cc:1468] Processing CreateTablet for tablet c0f8e222d6aa4f5582688f874ce04f4f (DEFAULT_TABLE table=default.Table_7c37f5cca4df4a929018f7a81e5cdcb3 [id=6d7c623de5b041e6b8ac74486e0c514b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.656539 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f8e222d6aa4f5582688f874ce04f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.656862 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 5af2b974df3a41f7b60537c9d8eac1e6 (DEFAULT_TABLE table=default.Table_f6b4ad595d094974adb62da4fe761a46 [id=df7316cd20aa497c9ad70964acb1b03f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.657196 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af2b974df3a41f7b60537c9d8eac1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.659212 1166 tablet_bootstrap.cc:654] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.659389 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 197e33dc3bad45cb9db7286b472a7111 (DEFAULT_TABLE table=default.Table_af1cd42de11e4022ac9d9550faca59b6 [id=64915e8dad4946e1bd552eba5b285c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.659723 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197e33dc3bad45cb9db7286b472a7111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.663158 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d361e73fe3d4ea2bb6735f814345ebb with delete_type TABLET_DATA_DELETED (replaced by tablet 50ac647b1d704142b94da8d506319466) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.664738 757 tablet_service.cc:1468] Processing CreateTablet for tablet 4616015b587a459abfb5403b690d23cf (DEFAULT_TABLE table=default.Table_2a0fc7dfe9884d9eb17952b2d9d3d5ee [id=612c57f12aef41cea072c4ae51552545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.665059 1170 tablet_bootstrap.cc:654] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.665707 756 tablet_service.cc:1468] Processing CreateTablet for tablet 156da2a965c64d1090a4f9d4eafdec70 (DEFAULT_TABLE table=default.Table_67a5cc5e1f4449029550030a01425b01 [id=04c08abdf697435d93d32751addfdd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.666042 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156da2a965c64d1090a4f9d4eafdec70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.666006 740 tablet_service.cc:1468] Processing CreateTablet for tablet d3a17198b9a04138b24022cf8506ab36 (DEFAULT_TABLE table=default.Table_1f52a2da8b374d99952264989d3f5b90 [id=4d086dd8cf994bb4ae2cdd7ca7389054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.665684 741 tablet_service.cc:1468] Processing CreateTablet for tablet 5e13a05108d545629f0a5e206dca270c (DEFAULT_TABLE table=default.Table_97642fc925bc492e9be22bec273710f0 [id=f75dad2552f846a297fcc0ea3b622d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.666286 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a17198b9a04138b24022cf8506ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.666484 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e13a05108d545629f0a5e206dca270c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.666738 744 tablet_service.cc:1468] Processing CreateTablet for tablet 171ce24a435e40e48026039941ca7883 (DEFAULT_TABLE table=default.Table_2c513d91226c4bf6aa6643cc25131331 [id=56c8e8e69f08428887e7f120dd4f405d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.667032 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171ce24a435e40e48026039941ca7883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.669389 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d3a17198b9a04138b24022cf8506ab36 (DEFAULT_TABLE table=default.Table_1f52a2da8b374d99952264989d3f5b90 [id=4d086dd8cf994bb4ae2cdd7ca7389054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.669796 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a17198b9a04138b24022cf8506ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.672972 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 171ce24a435e40e48026039941ca7883 (DEFAULT_TABLE table=default.Table_2c513d91226c4bf6aa6643cc25131331 [id=56c8e8e69f08428887e7f120dd4f405d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.673341 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171ce24a435e40e48026039941ca7883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.677546 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4616015b587a459abfb5403b690d23cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.683077 880 tablet_service.cc:1468] Processing CreateTablet for tablet 171ce24a435e40e48026039941ca7883 (DEFAULT_TABLE table=default.Table_2c513d91226c4bf6aa6643cc25131331 [id=56c8e8e69f08428887e7f120dd4f405d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.689069 884 tablet_service.cc:1468] Processing CreateTablet for tablet 156da2a965c64d1090a4f9d4eafdec70 (DEFAULT_TABLE table=default.Table_67a5cc5e1f4449029550030a01425b01 [id=04c08abdf697435d93d32751addfdd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.689759 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156da2a965c64d1090a4f9d4eafdec70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.691020 1165 ts_tablet_manager.cc:1918] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.691155 1165 log.cc:1199] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/01ba67b500b646d7aa55315f7ee23ed6
I20250625 13:52:03.691494 1165 ts_tablet_manager.cc:1939] T 01ba67b500b646d7aa55315f7ee23ed6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:03.692231 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 568) took 1244 ms. Trace:
I20250625 13:52:03.692328 1165 rpcz_store.cc:276] 0625 13:52:02.448013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.453128 (+ 5115us) service_pool.cc:225] Handling call
0625 13:52:03.611804 (+1158676us) ts_tablet_manager.cc:1145] Deleting tablet 01ba67b500b646d7aa55315f7ee23ed6
0625 13:52:03.611816 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.655018 (+ 43202us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.690925 (+ 35907us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.692209 (+ 1284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158456}
I20250625 13:52:03.692898 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 01ba67b500b646d7aa55315f7ee23ed6 (table default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]) successfully deleted
I20250625 13:52:03.693370 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5e13a05108d545629f0a5e206dca270c (DEFAULT_TABLE table=default.Table_97642fc925bc492e9be22bec273710f0 [id=f75dad2552f846a297fcc0ea3b622d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.693853 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e13a05108d545629f0a5e206dca270c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.695607 892 tablet_service.cc:1468] Processing CreateTablet for tablet 4616015b587a459abfb5403b690d23cf (DEFAULT_TABLE table=default.Table_2a0fc7dfe9884d9eb17952b2d9d3d5ee [id=612c57f12aef41cea072c4ae51552545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.695984 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4616015b587a459abfb5403b690d23cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.698901 887 tablet_service.cc:1468] Processing CreateTablet for tablet d3a17198b9a04138b24022cf8506ab36 (DEFAULT_TABLE table=default.Table_1f52a2da8b374d99952264989d3f5b90 [id=4d086dd8cf994bb4ae2cdd7ca7389054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.700196 1710 ts_tablet_manager.cc:1918] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.700630 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 156da2a965c64d1090a4f9d4eafdec70 (DEFAULT_TABLE table=default.Table_67a5cc5e1f4449029550030a01425b01 [id=04c08abdf697435d93d32751addfdd8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.701496 889 tablet_service.cc:1468] Processing CreateTablet for tablet 5af2b974df3a41f7b60537c9d8eac1e6 (DEFAULT_TABLE table=default.Table_f6b4ad595d094974adb62da4fe761a46 [id=df7316cd20aa497c9ad70964acb1b03f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.702042 888 tablet_service.cc:1468] Processing CreateTablet for tablet c0f8e222d6aa4f5582688f874ce04f4f (DEFAULT_TABLE table=default.Table_7c37f5cca4df4a929018f7a81e5cdcb3 [id=6d7c623de5b041e6b8ac74486e0c514b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.703070 891 tablet_service.cc:1468] Processing CreateTablet for tablet 197e33dc3bad45cb9db7286b472a7111 (DEFAULT_TABLE table=default.Table_af1cd42de11e4022ac9d9550faca59b6 [id=64915e8dad4946e1bd552eba5b285c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.703583 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a17198b9a04138b24022cf8506ab36. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:03.704010 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 60ef50a3624b4c79acc803ff0e2ed364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ef50a3624b4c79acc803ff0e2ed364 already in progress: creating tablet
W20250625 13:52:03.704751 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 39540b81fd4746858e65be49ded0e1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39540b81fd4746858e65be49ded0e1df already in progress: creating tablet
W20250625 13:52:03.705344 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 57afeff143c846289c6159974b00d7ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57afeff143c846289c6159974b00d7ba already in progress: creating tablet
I20250625 13:52:03.705691 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 572) took 1002 ms. Trace:
I20250625 13:52:03.706029 1165 rpcz_store.cc:276] 0625 13:52:02.703726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.743203 (+ 39477us) service_pool.cc:225] Handling call
0625 13:52:03.705597 (+962394us) ts_tablet_manager.cc:1145] Deleting tablet 2c8cf5d6cd624a1797486c69bd2098fd
0625 13:52:03.705604 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.705673 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926877}
W20250625 13:52:03.706678 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2c8cf5d6cd624a1797486c69bd2098fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8cf5d6cd624a1797486c69bd2098fd already in progress: creating tablet
W20250625 13:52:03.707268 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d2f545a5934748b2af767d37ea937fca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2f545a5934748b2af767d37ea937fca already in progress: creating tablet
I20250625 13:52:03.707464 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171ce24a435e40e48026039941ca7883. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.707608 1165 tablet_replica.cc:331] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:03.707768 1165 raft_consensus.cc:2241] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.707927 1165 raft_consensus.cc:2270] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.708495 1165 ts_tablet_manager.cc:1905] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.710387 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156da2a965c64d1090a4f9d4eafdec70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.711428 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f8e222d6aa4f5582688f874ce04f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.712782 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af2b974df3a41f7b60537c9d8eac1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.722962 1748 raft_consensus.cc:491] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c1aa07ab6bc4b0089e46951240a37c9)
I20250625 13:52:03.724431 1748 raft_consensus.cc:513] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.725275 1748 leader_election.cc:290] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:03.726029 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197e33dc3bad45cb9db7286b472a7111. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:03.726624 851 leader_election.cc:343] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 583ee2f2c66841939c93104efc36facb
I20250625 13:52:03.727217 1710 log.cc:1199] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/1f6be7189f8f45ccb329ef5d2fd8da8d
W20250625 13:52:03.728814 849 leader_election.cc:343] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 583ee2f2c66841939c93104efc36facb
I20250625 13:52:03.728976 849 leader_election.cc:304] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:03.729481 1748 raft_consensus.cc:2747] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:03.756639 1710 ts_tablet_manager.cc:1939] T 1f6be7189f8f45ccb329ef5d2fd8da8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:03.757460 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e29bade90b184d2483cb1fe85033a03c consensus state has no leader
I20250625 13:52:03.758366 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 519) took 1990 ms. Trace:
I20250625 13:52:03.758513 1710 rpcz_store.cc:276] 0625 13:52:01.766815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.766862 (+ 47us) service_pool.cc:225] Handling call
0625 13:52:03.376316 (+1609454us) ts_tablet_manager.cc:1145] Deleting tablet 1f6be7189f8f45ccb329ef5d2fd8da8d
0625 13:52:03.376324 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.550047 (+173723us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.691499 (+141452us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.757176 (+ 65677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1609276}
I20250625 13:52:03.759030 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 1f6be7189f8f45ccb329ef5d2fd8da8d (table default.Table_1be3d9db61ab4b56b98b70af09df98b3 [id=68e44a6915224107b42ca15194e2fed9]) successfully deleted
I20250625 13:52:03.761318 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7c6d7f469e1044c8b0696ca89aac611e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.762048 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c6d7f469e1044c8b0696ca89aac611e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.764503 1171 tablet_bootstrap.cc:654] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:03.765255 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 288116us
I20250625 13:52:03.768029 1803 consensus_queue.cc:1035] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250625 13:52:03.770325 1710 tablet_replica.cc:331] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:03.770555 1710 raft_consensus.cc:2241] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.759372 1801 tablet.cc:1722] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:03.776511 1166 tablet_bootstrap.cc:492] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:03.776670 1166 ts_tablet_manager.cc:1397] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.238s user 0.020s sys 0.020s
I20250625 13:52:03.777349 1166 raft_consensus.cc:357] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.777523 1166 raft_consensus.cc:383] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.777593 1166 raft_consensus.cc:738] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.777834 1166 consensus_queue.cc:260] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.778893 1168 ts_tablet_manager.cc:1918] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.779038 1168 log.cc:1199] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/bb1c7a85da844d34be805843bc60d350
I20250625 13:52:03.780154 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3c6dd5b87ff34151836a3c29740e7c11
I20250625 13:52:03.781401 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c10378f96d2b4caea5653d98df60ef50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.781884 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7db450dc63bd4b6caf074c67490cd175
W20250625 13:52:03.781972 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c10378f96d2b4caea5653d98df60ef50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.782591 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_96a404a50c0641bf98518a2c816eaefa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.782887 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 455884b62ee44b6da53543093f721027
W20250625 13:52:03.783133 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96a404a50c0641bf98518a2c816eaefa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.783582 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c9810f2e24dc4b31a0a4fdb1af3ef7db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.784072 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9810f2e24dc4b31a0a4fdb1af3ef7db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.785018 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4d25e4114a184bc99705042593ab4ab9
I20250625 13:52:03.785660 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet be3313ec5dfb41b68662d3ff4c8c94f9
I20250625 13:52:03.786934 1165 ts_tablet_manager.cc:1918] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.787072 1165 log.cc:1199] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/f00e72e103f443e886f0df690cff4b6f
I20250625 13:52:03.787258 1776 raft_consensus.cc:491] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.787391 1776 raft_consensus.cc:513] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.787894 1776 leader_election.cc:290] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:03.789309 1166 ts_tablet_manager.cc:1428] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 13:52:03.789521 1168 ts_tablet_manager.cc:1939] T bb1c7a85da844d34be805843bc60d350 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:03.790370 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 271) took 7100 ms. Trace:
I20250625 13:52:03.791592 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 455884b62ee44b6da53543093f721027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.791862 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d25e4114a184bc99705042593ab4ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.792141 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a31f10c8ca745b7ad635c622a265450" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:03.792763 711 leader_election.cc:343] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.792114 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c6dd5b87ff34151836a3c29740e7c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.792273 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 455884b62ee44b6da53543093f721027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.792517 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d25e4114a184bc99705042593ab4ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.795780 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c6dd5b87ff34151836a3c29740e7c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.796628 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db450dc63bd4b6caf074c67490cd175 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.796818 880 tablet_service.cc:1515] Processing DeleteTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.792518 740 tablet_service.cc:1515] Processing DeleteTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.798076 1168 rpcz_store.cc:276] 0625 13:51:56.689984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.720050 (+ 30066us) service_pool.cc:225] Handling call
0625 13:52:03.565373 (+6845323us) ts_tablet_manager.cc:1145] Deleting tablet bb1c7a85da844d34be805843bc60d350
0625 13:52:03.565384 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.569380 (+ 3996us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.778749 (+209369us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.790333 (+ 11584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6682135}
I20250625 13:52:03.798481 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet bb1c7a85da844d34be805843bc60d350 (table default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]) successfully deleted
I20250625 13:52:03.798758 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a31f10c8ca745b7ad635c622a265450 with delete_type TABLET_DATA_DELETED (replaced by tablet df1ed922750f45eab02d1fc0c26af75e) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.799096 1166 tablet_bootstrap.cc:492] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:03.799820 1171 tablet_bootstrap.cc:492] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.799993 1171 ts_tablet_manager.cc:1397] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.187s user 0.002s sys 0.002s
I20250625 13:52:03.800814 1171 raft_consensus.cc:357] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.801002 1171 raft_consensus.cc:383] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.801083 1171 raft_consensus.cc:738] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.801826 1171 consensus_queue.cc:260] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.802353 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_eaf03b2c45a842c392e299f6fe84dc05" } new_table_name: "default.Table_f4de7823c10e40e1bdd42b0e0559f7bf" modify_external_catalogs: true
I20250625 13:52:03.802615 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db450dc63bd4b6caf074c67490cd175 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.803112 1171 ts_tablet_manager.cc:1428] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:03.803514 1171 tablet_bootstrap.cc:492] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:03.804234 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_002bf8bcbdb849c2988585953c8828b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.804688 1168 tablet_replica.cc:331] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:03.804833 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_002bf8bcbdb849c2988585953c8828b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.805006 1168 raft_consensus.cc:2241] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:03.805387 1168 raft_consensus.cc:2270] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.807060 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2211bebab65640d09cfecd34b8eaaa40" } new_table_name: "default.Table_2009356d754d48c9a01a3058fe1ac4c5" modify_external_catalogs: true
I20250625 13:52:03.808396 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d8fbb4df3d0745b6a9671d0dbfb1c789" } new_table_name: "default.Table_0add09be48364d3a9aebeb9664561d09" modify_external_catalogs: true
I20250625 13:52:03.802428 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_845a00e4f2b34e549af0d49c36c7899d" } new_table_name: "default.Table_3fe8ea719ca94aaca7b19fadc398d91d" modify_external_catalogs: true
I20250625 13:52:03.810170 1165 ts_tablet_manager.cc:1939] T f00e72e103f443e886f0df690cff4b6f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:03.810761 1710 raft_consensus.cc:2270] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.810896 1168 ts_tablet_manager.cc:1905] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:03.813102 851 consensus_peers.cc:487] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0c5bea3b6214fa8af2792c5e9c411d9. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:03.811259 1171 tablet_bootstrap.cc:654] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.814983 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet f00e72e103f443e886f0df690cff4b6f (table default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]) successfully deleted
I20250625 13:52:03.815016 1165 tablet_replica.cc:331] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:03.815188 1165 raft_consensus.cc:2241] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.815387 1165 raft_consensus.cc:2270] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.817065 1166 tablet_bootstrap.cc:654] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.818245 1165 ts_tablet_manager.cc:1905] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.820206 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 with delete_type TABLET_DATA_DELETED (replaced by tablet 86af0ea6648045a1ac4ba903d20849b7) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.829015 1171 tablet_bootstrap.cc:492] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.829315 1171 ts_tablet_manager.cc:1397] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250625 13:52:03.830371 1171 raft_consensus.cc:357] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.831049 1171 raft_consensus.cc:383] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.831184 1171 raft_consensus.cc:738] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.831655 1171 consensus_queue.cc:260] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.832445 1710 ts_tablet_manager.cc:1905] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.835127 1171 ts_tablet_manager.cc:1428] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250625 13:52:03.836100 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c6dd5b87ff34151836a3c29740e7c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.836104 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a31f10c8ca745b7ad635c622a265450" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:03.836349 1171 tablet_bootstrap.cc:492] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:03.836867 711 leader_election.cc:343] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.836997 711 leader_election.cc:304] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:03.837369 1776 raft_consensus.cc:2747] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.837924 983 leader_election.cc:304] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:03.838245 1171 tablet_bootstrap.cc:654] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:03.838331 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 72662.3us
I20250625 13:52:03.838657 983 leader_election.cc:304] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:03.838922 983 raft_consensus.cc:2684] T f73f420a4d7a438eadde3d6af64f0a11 P 1c008de91eb444f48da2c4f0b2d7a1ea: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:52:03.841063 1803 raft_consensus.cc:2802] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:03.841194 1803 raft_consensus.cc:491] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:03.841280 1803 raft_consensus.cc:3058] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:03.845705 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_73b8d45e1da448d481ef5bbb2191c6cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.846477 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73b8d45e1da448d481ef5bbb2191c6cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.838852 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 with delete_type TABLET_DATA_DELETED (replaced by tablet 32143e2497ee4d1b8c2e7cbfc6160ef4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.853659 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db450dc63bd4b6caf074c67490cd175 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.856434 1168 ts_tablet_manager.cc:1918] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:03.856911 1168 log.cc:1199] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/412a0c5f05ab489ab9f8fc3f44fa7bdc
I20250625 13:52:03.857416 1168 ts_tablet_manager.cc:1939] T 412a0c5f05ab489ab9f8fc3f44fa7bdc P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:03.858042 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dfaba56442c0498da95e6fad126ab86d consensus state has no leader
I20250625 13:52:03.858299 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 281) took 6923 ms. Trace:
W20250625 13:52:03.858428 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e280bd329764924a6fd42c591888ed0 consensus state has no leader
I20250625 13:52:03.858388 1168 rpcz_store.cc:276] 0625 13:51:56.934539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:56.975276 (+ 40737us) service_pool.cc:225] Handling call
0625 13:52:03.804624 (+6829348us) ts_tablet_manager.cc:1145] Deleting tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc
0625 13:52:03.804634 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.821266 (+ 16632us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.856297 (+ 35031us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.858241 (+ 1944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6666469}
W20250625 13:52:03.858531 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 consensus state has no leader
W20250625 13:52:03.858814 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd4142572b344456a0d8248be914ad8f consensus state has no leader
I20250625 13:52:03.859110 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 412a0c5f05ab489ab9f8fc3f44fa7bdc (table default.Table_a9b04cfcee6646909171bdaf821ff85e [id=e0d6b782ef7a4e1f877871acba829d24]) successfully deleted
I20250625 13:52:03.860395 1166 tablet_bootstrap.cc:492] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:03.861250 1166 ts_tablet_manager.cc:1397] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.001s
I20250625 13:52:03.862210 1166 raft_consensus.cc:357] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.863018 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 455884b62ee44b6da53543093f721027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.867771 880 tablet_service.cc:1468] Processing CreateTablet for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 (DEFAULT_TABLE table=default.Table_35b3cb8e94364083971a19c3e8bc98a9 [id=936030c2a97249b09040f0fcea2da621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.868170 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e216856d5a4a3f8cd3a3852ffe2c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.869319 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 282) took 6932 ms. Trace:
I20250625 13:52:03.869416 1168 rpcz_store.cc:276] 0625 13:51:56.937077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.138184 (+201107us) service_pool.cc:225] Handling call
0625 13:52:03.869124 (+6730940us) ts_tablet_manager.cc:1145] Deleting tablet 6c5fc63828d9408b9cf774a966f43cf2
0625 13:52:03.869133 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.869255 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6697493}
W20250625 13:52:03.872188 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6c5fc63828d9408b9cf774a966f43cf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c5fc63828d9408b9cf774a966f43cf2 already in progress: creating tablet
I20250625 13:52:03.876260 881 tablet_service.cc:1468] Processing CreateTablet for tablet b120e9ea0fc04b3dbce52cdc33a713d5 (DEFAULT_TABLE table=default.Table_74dfe85879a04d7eb5bee2e8f00065ab [id=050b47e9354146958edc009630ea65d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.876715 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b120e9ea0fc04b3dbce52cdc33a713d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.876772 891 tablet_service.cc:1468] Processing CreateTablet for tablet 882152a3a5a2458d803ba7f75b31bc30 (DEFAULT_TABLE table=default.Table_4190488f5e9f436d986974482c2e5ecd [id=198d682981a142d898f2d25ef848b2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.877156 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882152a3a5a2458d803ba7f75b31bc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.878003 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 286) took 6939 ms. Trace:
I20250625 13:52:03.878120 1168 rpcz_store.cc:276] 0625 13:51:56.938459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.171651 (+233192us) service_pool.cc:225] Handling call
0625 13:52:03.877825 (+6706174us) ts_tablet_manager.cc:1145] Deleting tablet 7ba1338d0562418eadabdb8dbcaedee2
0625 13:52:03.877833 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.877933 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6705946}
I20250625 13:52:03.878988 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 285) took 6940 ms. Trace:
I20250625 13:52:03.879096 1168 rpcz_store.cc:276] 0625 13:51:56.938330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.169647 (+231317us) service_pool.cc:225] Handling call
0625 13:52:03.878814 (+6709167us) ts_tablet_manager.cc:1145] Deleting tablet 29be8c0350584de382985e66556bdcca
0625 13:52:03.878824 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.878911 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6677487}
I20250625 13:52:03.879503 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 283) took 6942 ms. Trace:
I20250625 13:52:03.879592 888 tablet_service.cc:1468] Processing CreateTablet for tablet 43b791fd11914591bb812cc155d950dc (DEFAULT_TABLE table=default.Table_bd76ad95ad844003ad3afa39db18c7f8 [id=edd692bcb0914ed89d5daf5f5d73d285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.879945 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b791fd11914591bb812cc155d950dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.880563 1171 tablet_bootstrap.cc:492] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.880803 1171 ts_tablet_manager.cc:1397] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20250625 13:52:03.881731 1171 raft_consensus.cc:357] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.882045 1171 raft_consensus.cc:383] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.882206 1171 raft_consensus.cc:738] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.882547 1171 consensus_queue.cc:260] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.883641 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9710609da88d4b42993b3aef02416543 (DEFAULT_TABLE table=default.Table_d67dfdd23e284e7588f0d5fade2e36e7 [id=15331889813d404d9f15fe761fd6642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.884092 884 tablet_service.cc:1468] Processing CreateTablet for tablet bf035c7c4900419e9d26b9a2d81fded3 (DEFAULT_TABLE table=default.Table_ae4a88bee59a4776b7bf5b7fb6b31721 [id=8951666f78d741e380f003c44a912f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.884497 889 tablet_service.cc:1468] Processing CreateTablet for tablet 2e2711256c9042ebb651ae0ca3d02c89 (DEFAULT_TABLE table=default.Table_21b38804564840edbbc09582cfe8f506 [id=2c58de4ccc254d78bd25ae90198ee701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.884866 887 tablet_service.cc:1468] Processing CreateTablet for tablet b70e7607709e4be395f1ed8d01d7d169 (DEFAULT_TABLE table=default.Table_583f49dbf61c498b98a3cd236437318d [id=dcba0cb206674a0e8a4e7d045aa5f40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.885262 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1696cbd156f94d14a5f1da76c8e58584 (DEFAULT_TABLE table=default.Table_7c55c56331834053a65654bb59304b61 [id=045a63f5e57847fdb5c31867f7860228]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.885671 896 tablet_service.cc:1468] Processing CreateTablet for tablet 9d99fb74666d4468880176db558ba1c8 (DEFAULT_TABLE table=default.Table_e03511e0ab134ad799672ced1a244df2 [id=a1d25b68ab71475eb99325a38190ae81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.885946 1171 ts_tablet_manager.cc:1428] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:03.886173 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9710609da88d4b42993b3aef02416543. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:03.893527 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 7ba1338d0562418eadabdb8dbcaedee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba1338d0562418eadabdb8dbcaedee2 already in progress: creating tablet
W20250625 13:52:03.893822 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 29be8c0350584de382985e66556bdcca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29be8c0350584de382985e66556bdcca already in progress: creating tablet
I20250625 13:52:03.894500 1171 tablet_bootstrap.cc:492] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:03.896322 1171 tablet_bootstrap.cc:654] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.897735 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70e7607709e4be395f1ed8d01d7d169. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.897936 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf035c7c4900419e9d26b9a2d81fded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.901365 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2711256c9042ebb651ae0ca3d02c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.905365 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d99fb74666d4468880176db558ba1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.905838 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1696cbd156f94d14a5f1da76c8e58584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.907577 1171 tablet_bootstrap.cc:492] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.907817 1171 ts_tablet_manager.cc:1397] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20250625 13:52:03.908751 1171 raft_consensus.cc:357] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.908993 1171 raft_consensus.cc:383] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.909076 1171 raft_consensus.cc:738] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.909338 1171 consensus_queue.cc:260] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.910468 1171 ts_tablet_manager.cc:1428] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:03.911345 1171 tablet_bootstrap.cc:492] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:03.912964 1171 tablet_bootstrap.cc:654] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:03.916915 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a31f10c8ca745b7ad635c622a265450 with delete_type TABLET_DATA_DELETED (replaced by tablet df1ed922750f45eab02d1fc0c26af75e) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.929255 1171 tablet_bootstrap.cc:492] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:03.929409 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.929569 1171 ts_tablet_manager.cc:1397] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20250625 13:52:03.930392 1171 raft_consensus.cc:357] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.930665 1171 raft_consensus.cc:383] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.930666 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d25e4114a184bc99705042593ab4ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:03.930794 1171 raft_consensus.cc:738] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.931042 1171 consensus_queue.cc:260] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.932457 1171 ts_tablet_manager.cc:1428] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:03.932966 1171 tablet_bootstrap.cc:492] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:03.935057 1171 tablet_bootstrap.cc:654] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:03.936050 1811 log.cc:927] Time spent T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250625 13:52:03.864182 1165 ts_tablet_manager.cc:1918] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:03.936650 1165 log.cc:1199] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/27092991fcad475fabc77d1f8bab0582
I20250625 13:52:03.936996 1165 ts_tablet_manager.cc:1939] T 27092991fcad475fabc77d1f8bab0582 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:03.938463 1165 tablet_replica.cc:331] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:03.938639 1165 raft_consensus.cc:2241] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:03.938776 1165 raft_consensus.cc:2270] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:03.867223 752 tablet_service.cc:1468] Processing CreateTablet for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 (DEFAULT_TABLE table=default.Table_35b3cb8e94364083971a19c3e8bc98a9 [id=936030c2a97249b09040f0fcea2da621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.939201 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e216856d5a4a3f8cd3a3852ffe2c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.941358 1165 ts_tablet_manager.cc:1905] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:03.867920 746 tablet_service.cc:1468] Processing CreateTablet for tablet 882152a3a5a2458d803ba7f75b31bc30 (DEFAULT_TABLE table=default.Table_4190488f5e9f436d986974482c2e5ecd [id=198d682981a142d898f2d25ef848b2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.943645 887 tablet_service.cc:1515] Processing DeleteTablet for tablet f00e72e103f443e886f0df690cff4b6f with delete_type TABLET_DATA_DELETED (replaced by tablet a0e81d4c904248eba858462bcf31c5d9) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:03.943956 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882152a3a5a2458d803ba7f75b31bc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.946040 1024 tablet_service.cc:1468] Processing CreateTablet for tablet bf035c7c4900419e9d26b9a2d81fded3 (DEFAULT_TABLE table=default.Table_ae4a88bee59a4776b7bf5b7fb6b31721 [id=8951666f78d741e380f003c44a912f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.868363 740 tablet_service.cc:1468] Processing CreateTablet for tablet 43b791fd11914591bb812cc155d950dc (DEFAULT_TABLE table=default.Table_bd76ad95ad844003ad3afa39db18c7f8 [id=edd692bcb0914ed89d5daf5f5d73d285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.947568 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b791fd11914591bb812cc155d950dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.950909 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 9d99fb74666d4468880176db558ba1c8 (DEFAULT_TABLE table=default.Table_e03511e0ab134ad799672ced1a244df2 [id=a1d25b68ab71475eb99325a38190ae81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.869164 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9710609da88d4b42993b3aef02416543 (DEFAULT_TABLE table=default.Table_d67dfdd23e284e7588f0d5fade2e36e7 [id=15331889813d404d9f15fe761fd6642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.951584 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9710609da88d4b42993b3aef02416543. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.952886 1012 tablet_service.cc:1468] Processing CreateTablet for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 (DEFAULT_TABLE table=default.Table_35b3cb8e94364083971a19c3e8bc98a9 [id=936030c2a97249b09040f0fcea2da621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.953024 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 9710609da88d4b42993b3aef02416543 (DEFAULT_TABLE table=default.Table_d67dfdd23e284e7588f0d5fade2e36e7 [id=15331889813d404d9f15fe761fd6642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.953270 1020 tablet_service.cc:1468] Processing CreateTablet for tablet b70e7607709e4be395f1ed8d01d7d169 (DEFAULT_TABLE table=default.Table_583f49dbf61c498b98a3cd236437318d [id=dcba0cb206674a0e8a4e7d045aa5f40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.954505 1011 tablet_service.cc:1468] Processing CreateTablet for tablet b120e9ea0fc04b3dbce52cdc33a713d5 (DEFAULT_TABLE table=default.Table_74dfe85879a04d7eb5bee2e8f00065ab [id=050b47e9354146958edc009630ea65d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.870124 739 tablet_service.cc:1468] Processing CreateTablet for tablet bf035c7c4900419e9d26b9a2d81fded3 (DEFAULT_TABLE table=default.Table_ae4a88bee59a4776b7bf5b7fb6b31721 [id=8951666f78d741e380f003c44a912f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.870873 741 tablet_service.cc:1468] Processing CreateTablet for tablet 2e2711256c9042ebb651ae0ca3d02c89 (DEFAULT_TABLE table=default.Table_21b38804564840edbbc09582cfe8f506 [id=2c58de4ccc254d78bd25ae90198ee701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.871174 744 tablet_service.cc:1468] Processing CreateTablet for tablet b120e9ea0fc04b3dbce52cdc33a713d5 (DEFAULT_TABLE table=default.Table_74dfe85879a04d7eb5bee2e8f00065ab [id=050b47e9354146958edc009630ea65d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.955945 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf035c7c4900419e9d26b9a2d81fded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.956118 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b120e9ea0fc04b3dbce52cdc33a713d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.873149 743 tablet_service.cc:1468] Processing CreateTablet for tablet b70e7607709e4be395f1ed8d01d7d169 (DEFAULT_TABLE table=default.Table_583f49dbf61c498b98a3cd236437318d [id=dcba0cb206674a0e8a4e7d045aa5f40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.959571 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70e7607709e4be395f1ed8d01d7d169. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.961123 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 27092991fcad475fabc77d1f8bab0582 (table default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba [id=540530d4619a4093b20d1bcba9ba99ac]) successfully deleted
I20250625 13:52:03.961138 1170 tablet_bootstrap.cc:492] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:03.961936 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3016763b9e294b9da2861fe97e57e241 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3016763b9e294b9da2861fe97e57e241 already in progress: creating tablet
W20250625 13:52:03.962184 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 93fdde7d068b400ba7070166eff63502 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93fdde7d068b400ba7070166eff63502 already in progress: creating tablet
W20250625 13:52:03.962371 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c0f461a7a7ef45009279716a3024ee26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f461a7a7ef45009279716a3024ee26 already in progress: creating tablet
W20250625 13:52:03.962555 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0e77736191544e8da32ca9927f0a5804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e77736191544e8da32ca9927f0a5804 already in progress: creating tablet
W20250625 13:52:03.962723 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f197ea67c34546a7981bcd9df94702e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f197ea67c34546a7981bcd9df94702e6 already in progress: creating tablet
I20250625 13:52:03.963120 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 2e2711256c9042ebb651ae0ca3d02c89 (DEFAULT_TABLE table=default.Table_21b38804564840edbbc09582cfe8f506 [id=2c58de4ccc254d78bd25ae90198ee701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.873193 756 tablet_service.cc:1468] Processing CreateTablet for tablet 1696cbd156f94d14a5f1da76c8e58584 (DEFAULT_TABLE table=default.Table_7c55c56331834053a65654bb59304b61 [id=045a63f5e57847fdb5c31867f7860228]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.963649 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1696cbd156f94d14a5f1da76c8e58584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.963996 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 1696cbd156f94d14a5f1da76c8e58584 (DEFAULT_TABLE table=default.Table_7c55c56331834053a65654bb59304b61 [id=045a63f5e57847fdb5c31867f7860228]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.873445 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9d99fb74666d4468880176db558ba1c8 (DEFAULT_TABLE table=default.Table_e03511e0ab134ad799672ced1a244df2 [id=a1d25b68ab71475eb99325a38190ae81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.967562 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d99fb74666d4468880176db558ba1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.971407 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2711256c9042ebb651ae0ca3d02c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.972261 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf035c7c4900419e9d26b9a2d81fded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:03.962313 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f728647d61a4414ca11ff2b890235012" } new_table_name: "default.Table_9516364563e3471dbcbf1541c7ff5bd1" modify_external_catalogs: true
I20250625 13:52:03.975749 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 43b791fd11914591bb812cc155d950dc (DEFAULT_TABLE table=default.Table_bd76ad95ad844003ad3afa39db18c7f8 [id=edd692bcb0914ed89d5daf5f5d73d285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.976334 1166 raft_consensus.cc:383] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:03.976312 1026 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42238 (request call id 600) took 1134 ms. Trace:
I20250625 13:52:03.976413 1166 raft_consensus.cc:738] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:03.976606 1166 consensus_queue.cc:260] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.976873 1803 raft_consensus.cc:513] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:03.978143 1166 ts_tablet_manager.cc:1428] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.117s user 0.001s sys 0.001s
I20250625 13:52:03.978391 1166 tablet_bootstrap.cc:492] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:03.978642 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 882152a3a5a2458d803ba7f75b31bc30 (DEFAULT_TABLE table=default.Table_4190488f5e9f436d986974482c2e5ecd [id=198d682981a142d898f2d25ef848b2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.983016 1170 ts_tablet_manager.cc:1397] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.509s user 0.009s sys 0.027s
I20250625 13:52:03.983855 1170 raft_consensus.cc:357] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.984104 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d7704b38144429b9c95675be84a4730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:03.987136 1166 tablet_bootstrap.cc:654] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:03.986565 985 leader_election.cc:343] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 04479dfb34584f0891c604ef72bf93b5
I20250625 13:52:03.989688 1748 raft_consensus.cc:491] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:03.989871 1748 raft_consensus.cc:513] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:03.990479 1748 leader_election.cc:290] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:03.992101 851 leader_election.cc:343] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f00e72e103f443e886f0df690cff4b6f
I20250625 13:52:03.992297 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ddd89f53a2204d2f879f8fd45e20a888" } modify_external_catalogs: true
I20250625 13:52:03.992374 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ebff453083b745618b11918f62df6a0a" } modify_external_catalogs: true
I20250625 13:52:03.992523 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ddd89f53a2204d2f879f8fd45e20a888" } modify_external_catalogs: true
I20250625 13:52:03.992583 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ebff453083b745618b11918f62df6a0a" } modify_external_catalogs: true
I20250625 13:52:03.992969 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4145b0b039bc4d73a22b72a6340b02c5" } modify_external_catalogs: true
I20250625 13:52:03.993145 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4145b0b039bc4d73a22b72a6340b02c5" } modify_external_catalogs: true
I20250625 13:52:03.993121 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_88d04eecb5a14feda8d8662393261503"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.993381 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8082f0bf0ab7432da22e85d6b82c51f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:03.993606 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8cdfd947725147e294c585373c3c4fc7" } modify_external_catalogs: true
W20250625 13:52:03.993728 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88d04eecb5a14feda8d8662393261503 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.993775 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8cdfd947725147e294c585373c3c4fc7" } modify_external_catalogs: true
W20250625 13:52:03.993860 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8082f0bf0ab7432da22e85d6b82c51f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.994292 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a40662133d63412ea32d88ad295176a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.994719 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a40662133d63412ea32d88ad295176a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:03.995659 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fac2050c0ed04e3da584967a45644198" } modify_external_catalogs: true
I20250625 13:52:03.995808 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fac2050c0ed04e3da584967a45644198" } modify_external_catalogs: true
I20250625 13:52:03.998396 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f00e72e103f443e886f0df690cff4b6f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:03.998337 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a443a849fceb423c9b889f5c3c69b2b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:03.999673 849 leader_election.cc:343] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:03.999853 849 leader_election.cc:304] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:04.000079 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a443a849fceb423c9b889f5c3c69b2b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.000317 1748 raft_consensus.cc:2747] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:03.989914 1803 leader_election.cc:290] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.005614 1748 raft_consensus.cc:491] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.005815 1748 raft_consensus.cc:513] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.006438 1748 leader_election.cc:290] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.007068 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1acc41a93f6433588727811903921fe" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:04.007246 778 raft_consensus.cc:2466] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:04.007695 851 leader_election.cc:304] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:04.008412 1748 raft_consensus.cc:2802] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:04.008502 1748 raft_consensus.cc:491] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:04.008587 1748 raft_consensus.cc:3058] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.011827 1748 raft_consensus.cc:513] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.013151 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1acc41a93f6433588727811903921fe" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:04.013315 778 raft_consensus.cc:3058] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.013370 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_272f40fffdd4480784ee11e2b9fabc45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.014005 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_272f40fffdd4480784ee11e2b9fabc45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.015877 778 raft_consensus.cc:2466] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:04.016450 851 leader_election.cc:304] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:04.016721 1770 raft_consensus.cc:2802] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:04.016880 1770 raft_consensus.cc:695] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:04.017131 1770 consensus_queue.cc:237] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.020191 1748 leader_election.cc:290] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.020202 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1acc41a93f6433588727811903921fe" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:04.020988 849 leader_election.cc:343] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.029268 1170 raft_consensus.cc:383] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.029388 1170 raft_consensus.cc:738] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.029623 1170 consensus_queue.cc:260] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.030177 1170 ts_tablet_manager.cc:1428] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20250625 13:52:04.030225 1166 tablet_bootstrap.cc:492] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.030682 1026 rpcz_store.cc:276] 0625 13:52:02.832480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.293501 (+461021us) service_pool.cc:225] Handling call
0625 13:52:03.366261 (+ 72760us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0625 13:52:03.366287 (+ 26us) ts_tablet_manager.cc:676] Creating new metadata...
0625 13:52:03.697277 (+330990us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:03.966313 (+269036us) tablet_replica.cc:206] Creating consensus instance
0625 13:52:03.966470 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:52:04.032258 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d99fb74666d4468880176db558ba1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.032414 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1acc41a93f6433588727811903921fe" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.032541 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70e7607709e4be395f1ed8d01d7d169. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.034978 1166 ts_tablet_manager.cc:1397] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.006s
I20250625 13:52:04.037523 1776 raft_consensus.cc:491] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.037662 1776 raft_consensus.cc:513] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.038221 1776 leader_election.cc:290] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.038877 1710 ts_tablet_manager.cc:1918] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:04.038944 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d7704b38144429b9c95675be84a4730" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.039229 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9710609da88d4b42993b3aef02416543. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:04.039474 711 leader_election.cc:343] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:04.039680 849 leader_election.cc:343] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:04.039958 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f56a8a6cb964d3a979e2f1d944eeddc consensus state has no leader
I20250625 13:52:04.040005 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d7704b38144429b9c95675be84a4730" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:04.039019 1710 log.cc:1199] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/93f449c80b604a6498f6fd1d1ba4770a
I20250625 13:52:04.040127 1166 raft_consensus.cc:357] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.040300 1166 raft_consensus.cc:383] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.040362 1166 raft_consensus.cc:738] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.040716 1710 ts_tablet_manager.cc:1939] T 93f449c80b604a6498f6fd1d1ba4770a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:04.040628 1166 consensus_queue.cc:260] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.041098 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 520) took 2273 ms. Trace:
I20250625 13:52:04.041240 1710 rpcz_store.cc:276] 0625 13:52:01.767228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.767268 (+ 40us) service_pool.cc:225] Handling call
0625 13:52:03.759278 (+1992010us) ts_tablet_manager.cc:1145] Deleting tablet 93f449c80b604a6498f6fd1d1ba4770a
0625 13:52:03.759290 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.989076 (+229786us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.038724 (+ 49648us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.041054 (+ 2330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991788}
I20250625 13:52:04.041571 1710 tablet_replica.cc:331] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:04.042402 1166 ts_tablet_manager.cc:1428] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:04.042673 1166 tablet_bootstrap.cc:492] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.043450 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2711256c9042ebb651ae0ca3d02c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.044332 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882152a3a5a2458d803ba7f75b31bc30. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:04.044400 711 leader_election.cc:343] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.044513 711 leader_election.cc:304] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:04.044737 1776 raft_consensus.cc:2747] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.044822 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b791fd11914591bb812cc155d950dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.045188 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2e216856d5a4a3f8cd3a3852ffe2c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.045851 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b120e9ea0fc04b3dbce52cdc33a713d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.046275 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1696cbd156f94d14a5f1da76c8e58584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.046736 1166 tablet_bootstrap.cc:654] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.049724 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 93f449c80b604a6498f6fd1d1ba4770a (table default.Table_7ccc261a6367449583a5c30495bfe2a9 [id=5436e1b638d14b00a66842a43cc5c085]) successfully deleted
I20250625 13:52:04.050420 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04479dfb34584f0891c604ef72bf93b5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:04.050626 912 raft_consensus.cc:3058] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.054299 912 raft_consensus.cc:2466] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:04.055153 1007 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 119) took 1802 ms. Trace:
I20250625 13:52:04.055269 1006 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 120) took 1774 ms. Trace:
I20250625 13:52:04.055372 1006 rpcz_store.cc:276] 0625 13:52:02.280922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.280980 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:02.394174 (+113194us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x61800000d518. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18f0508364 00007f18f05078a6 00007f1903b54b1d 00007f1903b54abc 00007f18ffd72f96 00007f18ffd736eb 00007f18ffce1125 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e
0625 13:52:02.875838 (+481664us) spinlock_profiling.cc:243] Waited 905 us on lock 0x616000125c30. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd29add9 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:02.895495 (+ 19657us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x616000125c30. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd29be9c 00007f18fd29b3ca 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8
0625 13:52:03.427819 (+532324us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x616000142c54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:03.614318 (+186499us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x61800015048c. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00000000009eeed0 00007f18ffce16d1 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:03.661666 (+ 47348us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x616000131e54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:04.055211 (+393545us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":774809984}
I20250625 13:52:04.059494 983 leader_election.cc:304] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.059620 1166 tablet_bootstrap.cc:492] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.059744 1166 ts_tablet_manager.cc:1397] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250625 13:52:04.060431 1166 raft_consensus.cc:357] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.061033 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_736e3a7bd9a44cfda965390534b95893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.061511 1816 raft_consensus.cc:2802] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:52:04.061823 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_736e3a7bd9a44cfda965390534b95893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.062790 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d50d71211aba4a888036f693f4c1336e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.064769 1166 raft_consensus.cc:383] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.064867 1166 raft_consensus.cc:738] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.065342 1166 consensus_queue.cc:260] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:04.065761 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d50d71211aba4a888036f693f4c1336e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.071574 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_103d81d6145e465b96d0044d4fd3d4f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.071848 1166 ts_tablet_manager.cc:1428] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250625 13:52:04.072140 1166 tablet_bootstrap.cc:492] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:04.072124 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_103d81d6145e465b96d0044d4fd3d4f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.073675 1166 tablet_bootstrap.cc:654] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.074213 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet cd4f0857f52449e1b26fde09c6c7c9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.080845 1518 raft_consensus.cc:491] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.081027 1518 raft_consensus.cc:513] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.081665 1166 tablet_bootstrap.cc:492] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.081766 1518 leader_election.cc:290] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.082464 1166 ts_tablet_manager.cc:1397] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
W20250625 13:52:04.082749 985 leader_election.cc:343] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7d361e73fe3d4ea2bb6735f814345ebb
I20250625 13:52:04.082952 1007 rpcz_store.cc:276] 0625 13:52:02.252548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.252755 (+ 207us) service_pool.cc:225] Handling call
0625 13:52:02.393992 (+141237us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x61800000d518. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18f0507c3d 00007f18f05084b7 00007f190513894d 00007f19051388de 00007f18ffd72ffc 00007f18ffd736eb 00007f18ffce1125 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e
0625 13:52:02.961443 (+567451us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x61800007348c. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd49f8be 00007f18fd29b071 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8
0625 13:52:03.427278 (+465835us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x616000142c54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:03.666390 (+239112us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x616000131e54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:04.055111 (+388721us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582827776}
I20250625 13:52:04.083673 1166 raft_consensus.cc:357] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.086768 1166 raft_consensus.cc:383] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.087000 1166 raft_consensus.cc:738] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.087424 1166 consensus_queue.cc:260] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.090068 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d361e73fe3d4ea2bb6735f814345ebb" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:04.090337 912 raft_consensus.cc:2466] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:04.088452 1165 ts_tablet_manager.cc:1918] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.090637 1165 log.cc:1199] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ae4a105e80c1460cbb21e1bdb9b356b1
I20250625 13:52:04.090807 983 leader_election.cc:304] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.091168 1518 raft_consensus.cc:2802] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:04.091284 1518 raft_consensus.cc:491] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:04.091403 1518 raft_consensus.cc:3058] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.091930 1166 ts_tablet_manager.cc:1428] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250625 13:52:04.093469 1010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 121) took 1716 ms. Trace:
I20250625 13:52:04.095773 1166 tablet_bootstrap.cc:492] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.100034 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ab663b7eba452c9f9775c0f568010b with delete_type TABLET_DATA_DELETED (replaced by tablet 155737a8d5604eb69524c4d6bea61550) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.092267 1165 ts_tablet_manager.cc:1939] T ae4a105e80c1460cbb21e1bdb9b356b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.107007 1165 tablet_replica.cc:331] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.107188 1165 raft_consensus.cc:2241] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.107336 1165 raft_consensus.cc:2270] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.104630 1166 tablet_bootstrap.cc:654] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.112236 591 catalog_manager.cc:5582] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:04.112710 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]). Expected version 1 got 0
I20250625 13:52:04.114272 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet af05fba6fe2042aab5dd47b30e4dfcf3 (table default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]). Expected version 4 got 0
I20250625 13:52:04.115162 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f0caf6cb5cc84df9b75703d4a12eb505" } new_table_name: "default.Table_3c1061dac55a4e97ad139950a7c8f302" modify_external_catalogs: true
I20250625 13:52:04.115554 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ae4a105e80c1460cbb21e1bdb9b356b1 (table default.Table_d18abaf5b02340fa82d3fee69699e1e3 [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]) successfully deleted
W20250625 13:52:04.115808 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 455884b62ee44b6da53543093f721027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 455884b62ee44b6da53543093f721027 already in progress: creating tablet
W20250625 13:52:04.116019 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4d25e4114a184bc99705042593ab4ab9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d25e4114a184bc99705042593ab4ab9 already in progress: creating tablet
W20250625 13:52:04.116225 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3c6dd5b87ff34151836a3c29740e7c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c6dd5b87ff34151836a3c29740e7c11 already in progress: creating tablet
W20250625 13:52:04.116441 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet be3313ec5dfb41b68662d3ff4c8c94f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be3313ec5dfb41b68662d3ff4c8c94f9 already in progress: creating tablet
I20250625 13:52:04.117161 743 tablet_service.cc:1468] Processing CreateTablet for tablet 973c483c87b84ea09b6c7d556e54b490 (DEFAULT_TABLE table=default.Table_002bf8bcbdb849c2988585953c8828b0 [id=3f841b395ab7474c98dc79979df6889d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.117589 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973c483c87b84ea09b6c7d556e54b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.118749 1166 tablet_bootstrap.cc:492] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.118886 1166 ts_tablet_manager.cc:1397] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20250625 13:52:04.119606 1166 raft_consensus.cc:357] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.119760 1166 raft_consensus.cc:383] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.119861 1166 raft_consensus.cc:738] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.120072 1166 consensus_queue.cc:260] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.121181 896 tablet_service.cc:1468] Processing CreateTablet for tablet e3eb1b7781bc4a5fb10e9ba7a34fb536 (DEFAULT_TABLE table=default.Table_c10378f96d2b4caea5653d98df60ef50 [id=9c282d659a4541228dfa562770815863]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.121724 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3eb1b7781bc4a5fb10e9ba7a34fb536. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.122298 889 tablet_service.cc:1468] Processing CreateTablet for tablet 973c483c87b84ea09b6c7d556e54b490 (DEFAULT_TABLE table=default.Table_002bf8bcbdb849c2988585953c8828b0 [id=3f841b395ab7474c98dc79979df6889d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.122681 1166 ts_tablet_manager.cc:1428] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:04.123013 1166 tablet_bootstrap.cc:492] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.123771 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973c483c87b84ea09b6c7d556e54b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.123847 888 tablet_service.cc:1468] Processing CreateTablet for tablet 5d2ce669ecd441939be8e8c71eea7bfa (DEFAULT_TABLE table=default.Table_c798e33a708c4d87b816a43d3a9d2d2b [id=ac51e8f08d4c401b9e5590583c2fcb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.124253 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2ce669ecd441939be8e8c71eea7bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.122748 884 tablet_service.cc:1468] Processing CreateTablet for tablet 99fd8aa1ff1443c29b9c734d896a3bf6 (DEFAULT_TABLE table=default.Table_cbdacccee68e4114ab2784c7bda953df [id=e1515a342f294401a7f9248163d14e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.125424 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd8aa1ff1443c29b9c734d896a3bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.125844 1166 tablet_bootstrap.cc:654] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.125063 891 tablet_service.cc:1468] Processing CreateTablet for tablet 00d4acd387c84d55834f632f41045862 (DEFAULT_TABLE table=default.Table_c9810f2e24dc4b31a0a4fdb1af3ef7db [id=bfeeae79c0bc48beb8c1c6d6cefd97d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.126256 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d4acd387c84d55834f632f41045862. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:04.133350 851 consensus_peers.cc:487] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf1664afbab647b3baeed6391196bed9. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:04.134560 893 tablet_service.cc:1468] Processing CreateTablet for tablet 909f4bb91e784b29acbbc9763e35ab03 (DEFAULT_TABLE table=default.Table_73b8d45e1da448d481ef5bbb2191c6cd [id=d511bc1389b84f9cbf0c107f42afc7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.135071 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909f4bb91e784b29acbbc9763e35ab03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.138255 1165 ts_tablet_manager.cc:1905] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.139899 1018 tablet_service.cc:1468] Processing CreateTablet for tablet e2ff8e226d34414aac0742c6395e4e72 (DEFAULT_TABLE table=default.Table_7c6d7f469e1044c8b0696ca89aac611e [id=a2e109f7f0e447b9bb2815ba7f9b60d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.140311 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ff8e226d34414aac0742c6395e4e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.141492 887 tablet_service.cc:1468] Processing CreateTablet for tablet e2ff8e226d34414aac0742c6395e4e72 (DEFAULT_TABLE table=default.Table_7c6d7f469e1044c8b0696ca89aac611e [id=a2e109f7f0e447b9bb2815ba7f9b60d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.142068 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ff8e226d34414aac0742c6395e4e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.144661 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 (DEFAULT_TABLE table=default.Table_96a404a50c0641bf98518a2c816eaefa [id=f018b65cb7954ba69e120a88c430cd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.145041 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.147594 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d8257f79a42247b48f2736a545362f44 (DEFAULT_TABLE table=default.Table_679758c0160345d48f8a3bfc5a1a042d [id=97d3f73d4cf64de1afc9c2397cdfa278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.147949 1171 tablet_bootstrap.cc:492] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.147984 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8257f79a42247b48f2736a545362f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.148267 1171 ts_tablet_manager.cc:1397] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.215s user 0.008s sys 0.012s
I20250625 13:52:04.148557 1166 tablet_bootstrap.cc:492] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.148672 1166 ts_tablet_manager.cc:1397] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250625 13:52:04.149086 1171 raft_consensus.cc:357] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.149274 1171 raft_consensus.cc:383] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.149341 1171 raft_consensus.cc:738] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.149382 1166 raft_consensus.cc:357] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.149524 1166 raft_consensus.cc:383] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.149591 1166 raft_consensus.cc:738] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.149575 1171 consensus_queue.cc:260] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.149765 1166 consensus_queue.cc:260] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.150085 1171 ts_tablet_manager.cc:1428] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:04.150182 1166 ts_tablet_manager.cc:1428] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 13:52:04.150413 1171 tablet_bootstrap.cc:492] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.151881 1166 tablet_bootstrap.cc:492] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.152808 881 tablet_service.cc:1468] Processing CreateTablet for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 (DEFAULT_TABLE table=default.Table_96a404a50c0641bf98518a2c816eaefa [id=f018b65cb7954ba69e120a88c430cd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.153167 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.098023 1170 tablet_bootstrap.cc:492] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.099073 1816 raft_consensus.cc:695] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:04.154764 884 tablet_service.cc:1468] Processing CreateTablet for tablet d8257f79a42247b48f2736a545362f44 (DEFAULT_TABLE table=default.Table_679758c0160345d48f8a3bfc5a1a042d [id=97d3f73d4cf64de1afc9c2397cdfa278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.155097 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8257f79a42247b48f2736a545362f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.157887 1170 tablet_bootstrap.cc:654] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.160595 892 tablet_service.cc:1468] Processing CreateTablet for tablet 7f3de20ac5f945bdb62535258880186f (DEFAULT_TABLE table=default.Table_50ec7255c1514670a780c11526d562a8 [id=cf2d90052ef640bd9d3303ffc3561e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.161706 1166 tablet_bootstrap.cc:654] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.162931 1165 ts_tablet_manager.cc:1918] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.162891 741 tablet_service.cc:1468] Processing CreateTablet for tablet e3eb1b7781bc4a5fb10e9ba7a34fb536 (DEFAULT_TABLE table=default.Table_c10378f96d2b4caea5653d98df60ef50 [id=9c282d659a4541228dfa562770815863]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.163055 1165 log.cc:1199] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/9a31f10c8ca745b7ad635c622a265450
I20250625 13:52:04.163357 1165 ts_tablet_manager.cc:1939] T 9a31f10c8ca745b7ad635c622a265450 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.163435 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3eb1b7781bc4a5fb10e9ba7a34fb536. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.164896 1165 tablet_replica.cc:331] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.165089 1165 raft_consensus.cc:2241] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.165238 1165 raft_consensus.cc:2270] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.167135 1165 ts_tablet_manager.cc:1905] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.169903 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3de20ac5f945bdb62535258880186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.104127 1518 raft_consensus.cc:513] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.118510 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 973c483c87b84ea09b6c7d556e54b490 (DEFAULT_TABLE table=default.Table_002bf8bcbdb849c2988585953c8828b0 [id=3f841b395ab7474c98dc79979df6889d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.118916 1014 tablet_service.cc:1468] Processing CreateTablet for tablet e3eb1b7781bc4a5fb10e9ba7a34fb536 (DEFAULT_TABLE table=default.Table_c10378f96d2b4caea5653d98df60ef50 [id=9c282d659a4541228dfa562770815863]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.119236 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 5d2ce669ecd441939be8e8c71eea7bfa (DEFAULT_TABLE table=default.Table_c798e33a708c4d87b816a43d3a9d2d2b [id=ac51e8f08d4c401b9e5590583c2fcb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.119621 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 99fd8aa1ff1443c29b9c734d896a3bf6 (DEFAULT_TABLE table=default.Table_cbdacccee68e4114ab2784c7bda953df [id=e1515a342f294401a7f9248163d14e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.123929 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 00d4acd387c84d55834f632f41045862 (DEFAULT_TABLE table=default.Table_c9810f2e24dc4b31a0a4fdb1af3ef7db [id=bfeeae79c0bc48beb8c1c6d6cefd97d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.173578 1770 raft_consensus.cc:491] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.127862 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 909f4bb91e784b29acbbc9763e35ab03 (DEFAULT_TABLE table=default.Table_73b8d45e1da448d481ef5bbb2191c6cd [id=d511bc1389b84f9cbf0c107f42afc7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.173776 1770 raft_consensus.cc:513] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.138486 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 7f3de20ac5f945bdb62535258880186f (DEFAULT_TABLE table=default.Table_50ec7255c1514670a780c11526d562a8 [id=cf2d90052ef640bd9d3303ffc3561e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.174568 1770 leader_election.cc:290] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:04.176043 851 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 36da1da539e24f55978d516873cb0b27
I20250625 13:52:04.177515 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5d2ce669ecd441939be8e8c71eea7bfa (DEFAULT_TABLE table=default.Table_c798e33a708c4d87b816a43d3a9d2d2b [id=ac51e8f08d4c401b9e5590583c2fcb8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.177878 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2ce669ecd441939be8e8c71eea7bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.178237 1165 ts_tablet_manager.cc:1918] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.178337 1165 log.cc:1199] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/5d7704b38144429b9c95675be84a4730
I20250625 13:52:04.178609 1165 ts_tablet_manager.cc:1939] T 5d7704b38144429b9c95675be84a4730 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.181250 1165 tablet_replica.cc:331] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.181411 1165 raft_consensus.cc:2241] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.181556 1165 raft_consensus.cc:2270] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.182822 1165 ts_tablet_manager.cc:1905] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.188985 1171 tablet_bootstrap.cc:654] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.194263 756 tablet_service.cc:1468] Processing CreateTablet for tablet 00d4acd387c84d55834f632f41045862 (DEFAULT_TABLE table=default.Table_c9810f2e24dc4b31a0a4fdb1af3ef7db [id=bfeeae79c0bc48beb8c1c6d6cefd97d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.195802 1165 ts_tablet_manager.cc:1918] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.197081 1165 log.cc:1199] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/34ab663b7eba452c9f9775c0f568010b
I20250625 13:52:04.197407 1165 ts_tablet_manager.cc:1939] T 34ab663b7eba452c9f9775c0f568010b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.195986 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d4acd387c84d55834f632f41045862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.200835 751 tablet_service.cc:1468] Processing CreateTablet for tablet 909f4bb91e784b29acbbc9763e35ab03 (DEFAULT_TABLE table=default.Table_73b8d45e1da448d481ef5bbb2191c6cd [id=d511bc1389b84f9cbf0c107f42afc7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.201333 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909f4bb91e784b29acbbc9763e35ab03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.203814 1171 tablet_bootstrap.cc:492] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.204053 1171 ts_tablet_manager.cc:1397] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.006s
I20250625 13:52:04.205109 1171 raft_consensus.cc:357] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.205540 1171 raft_consensus.cc:383] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.205646 1171 raft_consensus.cc:738] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.205894 1171 consensus_queue.cc:260] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.195523 741 tablet_service.cc:1468] Processing CreateTablet for tablet 99fd8aa1ff1443c29b9c734d896a3bf6 (DEFAULT_TABLE table=default.Table_cbdacccee68e4114ab2784c7bda953df [id=e1515a342f294401a7f9248163d14e52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:03.879592 1168 rpcz_store.cc:276] 0625 13:51:56.937225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.157645 (+220420us) service_pool.cc:225] Handling call
0625 13:52:03.879363 (+6721718us) ts_tablet_manager.cc:1145] Deleting tablet 46316eb725bc47e695af476c0b185504
0625 13:52:03.879372 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:03.879447 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6651029}
I20250625 13:52:04.213896 751 tablet_service.cc:1468] Processing CreateTablet for tablet 7f3de20ac5f945bdb62535258880186f (DEFAULT_TABLE table=default.Table_50ec7255c1514670a780c11526d562a8 [id=cf2d90052ef640bd9d3303ffc3561e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.214340 1171 ts_tablet_manager.cc:1428] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20250625 13:52:04.214620 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 46316eb725bc47e695af476c0b185504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46316eb725bc47e695af476c0b185504 already in progress: creating tablet
I20250625 13:52:04.215543 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3de20ac5f945bdb62535258880186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.215894 1171 tablet_bootstrap.cc:492] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.218178 1171 tablet_bootstrap.cc:654] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.229015 1166 tablet_bootstrap.cc:492] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.229223 1166 ts_tablet_manager.cc:1397] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.077s user 0.027s sys 0.013s
I20250625 13:52:04.230067 1166 raft_consensus.cc:357] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.230310 1166 raft_consensus.cc:383] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.230396 1166 raft_consensus.cc:738] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.230716 1166 consensus_queue.cc:260] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.233325 1166 ts_tablet_manager.cc:1428] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:04.233683 1166 tablet_bootstrap.cc:492] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.237316 1166 tablet_bootstrap.cc:654] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.238682 756 tablet_service.cc:1468] Processing CreateTablet for tablet e2ff8e226d34414aac0742c6395e4e72 (DEFAULT_TABLE table=default.Table_7c6d7f469e1044c8b0696ca89aac611e [id=a2e109f7f0e447b9bb2815ba7f9b60d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.244712 1776 raft_consensus.cc:491] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.244884 1776 raft_consensus.cc:513] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.245488 1776 leader_election.cc:290] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.246193 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af05fba6fe2042aab5dd47b30e4dfcf3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:04.246816 711 leader_election.cc:343] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.248349 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd8aa1ff1443c29b9c734d896a3bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.139874 1010 rpcz_store.cc:276] 0625 13:52:02.376766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.376838 (+ 72us) service_pool.cc:225] Handling call
0625 13:52:03.259638 (+882800us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x616000142c54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:03.552490 (+292852us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x616000111a54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:03.960233 (+407743us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x6160001b1430. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd29add9 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:04.093432 (+133199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":812791808}
I20250625 13:52:04.252470 744 tablet_service.cc:1468] Processing CreateTablet for tablet d8257f79a42247b48f2736a545362f44 (DEFAULT_TABLE table=default.Table_679758c0160345d48f8a3bfc5a1a042d [id=97d3f73d4cf64de1afc9c2397cdfa278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.252743 1171 tablet_bootstrap.cc:492] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.253051 1171 ts_tablet_manager.cc:1397] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20250625 13:52:04.253887 1171 raft_consensus.cc:357] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.254724 1171 raft_consensus.cc:383] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.254907 1171 raft_consensus.cc:738] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.255357 1171 consensus_queue.cc:260] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.256556 1171 ts_tablet_manager.cc:1428] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:04.154438 1816 consensus_queue.cc:237] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.257052 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3eb1b7781bc4a5fb10e9ba7a34fb536. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.257201 1171 tablet_bootstrap.cc:492] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.261656 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8257f79a42247b48f2736a545362f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.261969 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2ff8e226d34414aac0742c6395e4e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.266490 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d361e73fe3d4ea2bb6735f814345ebb" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:04.270408 912 raft_consensus.cc:3058] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.272439 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 (DEFAULT_TABLE table=default.Table_96a404a50c0641bf98518a2c816eaefa [id=f018b65cb7954ba69e120a88c430cd22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.272935 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.260629 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 284) took 7314 ms. Trace:
I20250625 13:52:04.273221 912 raft_consensus.cc:2466] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:04.264485 849 leader_election.cc:343] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 36da1da539e24f55978d516873cb0b27
I20250625 13:52:04.273869 849 leader_election.cc:304] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.274152 1510 raft_consensus.cc:2747] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:04.275748 1510 consensus_queue.cc:1035] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:04.276113 1171 tablet_bootstrap.cc:654] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.278956 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 9a31f10c8ca745b7ad635c622a265450 (table default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]) successfully deleted
I20250625 13:52:04.281513 1168 rpcz_store.cc:276] 0625 13:51:56.937333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.157778 (+220445us) service_pool.cc:225] Handling call
0625 13:52:04.236142 (+7078364us) ts_tablet_manager.cc:1145] Deleting tablet df1ed922750f45eab02d1fc0c26af75e
0625 13:52:04.236747 (+ 605us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.250315 (+ 13568us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6977341}
W20250625 13:52:04.286980 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet df1ed922750f45eab02d1fc0c26af75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df1ed922750f45eab02d1fc0c26af75e already in progress: creating tablet
I20250625 13:52:04.287521 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af05fba6fe2042aab5dd47b30e4dfcf3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:04.287591 985 leader_election.cc:343] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7d361e73fe3d4ea2bb6735f814345ebb
I20250625 13:52:04.287611 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 973c483c87b84ea09b6c7d556e54b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.289633 1518 leader_election.cc:290] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.291481 983 leader_election.cc:304] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.291955 1821 raft_consensus.cc:2802] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:04.292084 1821 raft_consensus.cc:695] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:04.292398 1821 consensus_queue.cc:237] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.293921 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 288) took 7350 ms. Trace:
I20250625 13:52:04.294086 1168 rpcz_store.cc:276] 0625 13:51:56.943224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.177357 (+234133us) service_pool.cc:225] Handling call
0625 13:52:04.293701 (+7116344us) ts_tablet_manager.cc:1145] Deleting tablet 11711546f0e14f9a9eabf003f49a69b7
0625 13:52:04.293716 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.293852 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7003383}
I20250625 13:52:04.295769 1168 tablet_replica.cc:331] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:04.296450 1168 raft_consensus.cc:2241] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.296933 1168 raft_consensus.cc:2270] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:04.297102 711 leader_election.cc:343] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.297255 711 leader_election.cc:304] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:04.297719 1776 raft_consensus.cc:2747] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.298468 1168 ts_tablet_manager.cc:1905] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:04.299196 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7db450dc63bd4b6caf074c67490cd175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db450dc63bd4b6caf074c67490cd175 already in progress: creating tablet
I20250625 13:52:04.299801 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 5d7704b38144429b9c95675be84a4730 (table default.Table_8c4176baf42b43658ec245c90d86cb98 [id=1786625fb4994b51a8b90c53e7e1714c]) successfully deleted
I20250625 13:52:04.300129 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 34ab663b7eba452c9f9775c0f568010b (table default.Table_e5291e5468c04390947d9ea217535cf6 [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
I20250625 13:52:04.306356 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e7a56419acce4220aed05859bbdad3bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.306928 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7a56419acce4220aed05859bbdad3bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.307011 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ae4a88bee59a4776b7bf5b7fb6b31721" } new_table_name: "default.Table_04e653ada53842b6b8e67c2b22879274" modify_external_catalogs: true
I20250625 13:52:04.308333 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0ff1884a92fe4a4490d4a3f5594fa263" } new_table_name: "default.Table_65fb980f16aa4d5d9aa270260fb1b896" modify_external_catalogs: true
I20250625 13:52:04.317325 1171 tablet_bootstrap.cc:492] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.317592 1171 ts_tablet_manager.cc:1397] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.005s
I20250625 13:52:04.319792 1171 raft_consensus.cc:357] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.320472 1171 raft_consensus.cc:383] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.320835 1171 raft_consensus.cc:738] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.329108 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b7062c34fd143bf86b743055a27db8c with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd84b8afa2548fe8ce5925f0bf806f0) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.329372 1165 tablet_replica.cc:331] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.329568 1165 raft_consensus.cc:2241] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.329704 1165 raft_consensus.cc:2270] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.329732 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 637847cac1724bfeb629c4d4013396ca with delete_type TABLET_DATA_DELETED (replaced by tablet b14d00c93a964edd815122e3723f89e7) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.335615 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet eabd67b9ceb74bd1a5f4f115c7db0e87
I20250625 13:52:04.337363 744 tablet_service.cc:1515] Processing DeleteTablet for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.338471 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0add09be48364d3a9aebeb9664561d09" } new_table_name: "default.Table_20e9351636684a39a1d996846e778bb4" modify_external_catalogs: true
I20250625 13:52:04.339598 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e3b3d5a2ab7042c2a2f18890363e7a9b
I20250625 13:52:04.339933 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 54fa042947da4e1b89018640b46b271a with delete_type TABLET_DATA_DELETED (replaced by tablet ab7202a8761444a0a731993d890fcdeb) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.340258 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_61c4552a00dd4dfb882f0e86f3b8776a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.340499 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:04.340843 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61c4552a00dd4dfb882f0e86f3b8776a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.341045 744 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b3d5a2ab7042c2a2f18890363e7a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:04.341456 581 consensus_peers.cc:487] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 -> Peer 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015): Couldn't send request to peer 3fec9d8a79054a5991ae90d8a3101e32. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 13:52:04.342317 591 master_service.cc:948] ReplaceTablet: received request to replace tablet eafc402d809245d189f3977162506ef8 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:04.342648 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d7660a2b70744bd589220b28e6edbc42
I20250625 13:52:04.343170 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 8fdca99b734448118e0c42227c78e58f from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:04.343688 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1b3d9a83b5544828a58cdd409a9417e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.344163 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b3d9a83b5544828a58cdd409a9417e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:04.345407 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 042359eacaa2451abc0fa6ed447a2905 consensus state has no leader
I20250625 13:52:04.345754 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2bcf5e3903254063bd1d49f7f9455737"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.345849 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d7660a2b70744bd589220b28e6edbc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:04.346146 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bcf5e3903254063bd1d49f7f9455737 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.352094 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b3d5a2ab7042c2a2f18890363e7a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.353472 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet d7660a2b70744bd589220b28e6edbc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.354048 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.357638 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c737daaede054d09a0c07564157423a9
I20250625 13:52:04.358428 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909f4bb91e784b29acbbc9763e35ab03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.358407 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fe7c9a0a684b4277a14d8ed100f3fa74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.358815 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd8aa1ff1443c29b9c734d896a3bf6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:04.358989 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe7c9a0a684b4277a14d8ed100f3fa74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.359788 1510 consensus_queue.cc:1035] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20250625 13:52:04.360908 1510 raft_consensus.cc:491] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.361939 1165 ts_tablet_manager.cc:1905] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.321872 1171 consensus_queue.cc:260] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.365842 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet c737daaede054d09a0c07564157423a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.371119 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet eafc402d809245d189f3977162506ef8
I20250625 13:52:04.373664 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet eafc402d809245d189f3977162506ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.374990 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a55db7a900494ec8bb877fd245f39f60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.375639 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a55db7a900494ec8bb877fd245f39f60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.375674 1770 raft_consensus.cc:491] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.376005 1748 raft_consensus.cc:491] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.376739 1510 raft_consensus.cc:513] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.377458 1510 leader_election.cc:290] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.378093 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "055f88e118454e32baa578cae9aceb0c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.379436 1710 raft_consensus.cc:2241] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:04.380409 1710 raft_consensus.cc:2270] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.381584 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2ce669ecd441939be8e8c71eea7bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.384410 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3de20ac5f945bdb62535258880186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.384907 1170 tablet_bootstrap.cc:492] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:04.385052 1170 ts_tablet_manager.cc:1397] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.355s user 0.023s sys 0.027s
I20250625 13:52:04.385558 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d4acd387c84d55834f632f41045862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.386025 1170 raft_consensus.cc:357] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.388093 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c737daaede054d09a0c07564157423a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.388890 743 tablet_service.cc:1515] Processing DeleteTablet for tablet eafc402d809245d189f3977162506ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.388376 1165 ts_tablet_manager.cc:1918] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.389299 1165 log.cc:1199] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/8b7062c34fd143bf86b743055a27db8c
I20250625 13:52:04.389616 1165 ts_tablet_manager.cc:1939] T 8b7062c34fd143bf86b743055a27db8c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.389932 1770 raft_consensus.cc:513] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:04.390148 849 leader_election.cc:343] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.390482 1748 raft_consensus.cc:513] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:04.391204 851 leader_election.cc:343] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 055f88e118454e32baa578cae9aceb0c
W20250625 13:52:04.391621 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf035c7c4900419e9d26b9a2d81fded3 consensus state has no leader
I20250625 13:52:04.391609 851 leader_election.cc:304] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.392014 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e2f0cb92dbc846c68be94207b9a9126b" } new_table_name: "default.Table_78be58e64e564e6ab48ccee86fbe5279" modify_external_catalogs: true
I20250625 13:52:04.392177 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 067272869c5547648c90f83a34a036aa from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:04.392354 1748 leader_election.cc:290] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:04.392608 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b61d4f7f0474aac9b46c532f6f403a0 consensus state has no leader
I20250625 13:52:04.392983 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8b535d0341e94ac2b1ea25b603b076ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.393131 1171 ts_tablet_manager.cc:1428] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.073s user 0.003s sys 0.000s
W20250625 13:52:04.393561 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b535d0341e94ac2b1ea25b603b076ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.393880 1748 raft_consensus.cc:2747] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.394109 1171 tablet_bootstrap.cc:492] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.394336 1770 leader_election.cc:290] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.396747 896 tablet_service.cc:1515] Processing DeleteTablet for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.409224 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39ac79102eab46688e767faed6d6f114" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.395831 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39ac79102eab46688e767faed6d6f114" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:04.408763 1170 raft_consensus.cc:383] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.410756 778 raft_consensus.cc:2466] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:04.412497 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 8b7062c34fd143bf86b743055a27db8c (table default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]) successfully deleted
W20250625 13:52:04.413357 851 leader_election.cc:343] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 27092991fcad475fabc77d1f8bab0582
I20250625 13:52:04.416543 1171 tablet_bootstrap.cc:654] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:04.419898 849 leader_election.cc:343] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.420274 851 leader_election.cc:304] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:04.422752 1770 raft_consensus.cc:2802] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:04.422864 1770 raft_consensus.cc:491] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:04.422961 1770 raft_consensus.cc:3058] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.425895 1770 raft_consensus.cc:513] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.426527 1770 leader_election.cc:290] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.428184 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39ac79102eab46688e767faed6d6f114" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:04.428371 778 raft_consensus.cc:3058] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.431102 778 raft_consensus.cc:2466] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:04.431766 1168 ts_tablet_manager.cc:1918] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.431736 851 leader_election.cc:304] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:04.432407 1748 raft_consensus.cc:2802] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:04.432543 1748 raft_consensus.cc:695] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:04.432860 1748 consensus_queue.cc:237] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.439568 1165 tablet_replica.cc:331] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.439848 1165 raft_consensus.cc:2241] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.440037 1165 raft_consensus.cc:2270] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.443990 1168 log.cc:1199] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/87327f74beb845768bbfd5ae6d2bcf88
I20250625 13:52:04.444828 1168 ts_tablet_manager.cc:1939] T 87327f74beb845768bbfd5ae6d2bcf88 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:04.445060 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27092991fcad475fabc77d1f8bab0582" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.446344 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 8ebe47e363ce4a6c8129169bc6777e63 from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:04.448307 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35aedb3b590d4917bac8fe0962d68fb3 consensus state has no leader
W20250625 13:52:04.445967 849 leader_election.cc:343] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.448036 896 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b3d5a2ab7042c2a2f18890363e7a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.455236 849 leader_election.cc:304] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.449110 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c737daaede054d09a0c07564157423a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.449608 892 tablet_service.cc:1515] Processing DeleteTablet for tablet eafc402d809245d189f3977162506ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.450322 889 tablet_service.cc:1468] Processing CreateTablet for tablet bdbf586b82194a24a2259cbeb62d615c (DEFAULT_TABLE table=default.Table_a40662133d63412ea32d88ad295176a6 [id=daebf8dbb2dd4d84b4c21c9a1770ee60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.455834 1748 raft_consensus.cc:2747] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.456127 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdbf586b82194a24a2259cbeb62d615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.456817 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 290) took 7241 ms. Trace:
I20250625 13:52:04.456944 1168 rpcz_store.cc:276] 0625 13:51:57.215600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.218325 (+ 2725us) service_pool.cc:225] Handling call
0625 13:52:04.295508 (+7077183us) ts_tablet_manager.cc:1145] Deleting tablet 87327f74beb845768bbfd5ae6d2bcf88
0625 13:52:04.295559 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.399860 (+104301us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.402996 (+ 3136us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.456764 (+ 53768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6995426}
I20250625 13:52:04.457301 1168 tablet_replica.cc:331] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:04.457312 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 consensus state has no leader
I20250625 13:52:04.459142 1023 tablet_service.cc:1468] Processing CreateTablet for tablet bdbf586b82194a24a2259cbeb62d615c (DEFAULT_TABLE table=default.Table_a40662133d63412ea32d88ad295176a6 [id=daebf8dbb2dd4d84b4c21c9a1770ee60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.451957 887 tablet_service.cc:1468] Processing CreateTablet for tablet e144ac4cd5e3415092856ff8e91016b3 (DEFAULT_TABLE table=default.Table_8082f0bf0ab7432da22e85d6b82c51f6 [id=7965625755b945fa9054b9cfa081aede]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.459621 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e144ac4cd5e3415092856ff8e91016b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.459878 1748 raft_consensus.cc:491] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.460036 1748 raft_consensus.cc:513] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:04.461728 851 leader_election.cc:343] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 46316eb725bc47e695af476c0b185504
I20250625 13:52:04.461890 1748 leader_election.cc:290] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.462533 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39ac79102eab46688e767faed6d6f114" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:04.462736 756 tablet_service.cc:1468] Processing CreateTablet for tablet bdbf586b82194a24a2259cbeb62d615c (DEFAULT_TABLE table=default.Table_a40662133d63412ea32d88ad295176a6 [id=daebf8dbb2dd4d84b4c21c9a1770ee60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.462831 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdbf586b82194a24a2259cbeb62d615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.463112 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdbf586b82194a24a2259cbeb62d615c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.463560 741 tablet_service.cc:1468] Processing CreateTablet for tablet e144ac4cd5e3415092856ff8e91016b3 (DEFAULT_TABLE table=default.Table_8082f0bf0ab7432da22e85d6b82c51f6 [id=7965625755b945fa9054b9cfa081aede]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.463881 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e144ac4cd5e3415092856ff8e91016b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.464926 1030 tablet_service.cc:1468] Processing CreateTablet for tablet d2972d4366c143498f746fd62a039acd (DEFAULT_TABLE table=default.Table_a443a849fceb423c9b889f5c3c69b2b6 [id=c2e68b746df2472c8a54cd070da4b73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.465266 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2972d4366c143498f746fd62a039acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.467569 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9a6bc49db6e741c2aa7762b202b3e1f5 (DEFAULT_TABLE table=default.Table_272f40fffdd4480784ee11e2b9fabc45 [id=62898d4eb2a84aa4975a78a7218a6a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.467895 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6bc49db6e741c2aa7762b202b3e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.471475 744 tablet_service.cc:1468] Processing CreateTablet for tablet c4db530b54f04fedb18bf993dc60d8da (DEFAULT_TABLE table=default.Table_88d04eecb5a14feda8d8662393261503 [id=ec3dfab61fff4a9ebb227df0e8816a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.471808 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4db530b54f04fedb18bf993dc60d8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.471720 743 tablet_service.cc:1468] Processing CreateTablet for tablet d2972d4366c143498f746fd62a039acd (DEFAULT_TABLE table=default.Table_a443a849fceb423c9b889f5c3c69b2b6 [id=c2e68b746df2472c8a54cd070da4b73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.472170 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2972d4366c143498f746fd62a039acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.452694 884 tablet_service.cc:1468] Processing CreateTablet for tablet d2972d4366c143498f746fd62a039acd (DEFAULT_TABLE table=default.Table_a443a849fceb423c9b889f5c3c69b2b6 [id=c2e68b746df2472c8a54cd070da4b73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.452890 881 tablet_service.cc:1468] Processing CreateTablet for tablet 9a6bc49db6e741c2aa7762b202b3e1f5 (DEFAULT_TABLE table=default.Table_272f40fffdd4480784ee11e2b9fabc45 [id=62898d4eb2a84aa4975a78a7218a6a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.453500 891 tablet_service.cc:1468] Processing CreateTablet for tablet c4db530b54f04fedb18bf993dc60d8da (DEFAULT_TABLE table=default.Table_88d04eecb5a14feda8d8662393261503 [id=ec3dfab61fff4a9ebb227df0e8816a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.475740 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2972d4366c143498f746fd62a039acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.479292 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4db530b54f04fedb18bf993dc60d8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.481587 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6bc49db6e741c2aa7762b202b3e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.481673 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 067272869c5547648c90f83a34a036aa
I20250625 13:52:04.482165 588 catalog_manager.cc:6652] ReplaceTablet: tablet 067272869c5547648c90f83a34a036aa deleted and replaced by tablet 6340de862fd04c12bb7f85557cfe53d2
I20250625 13:52:04.482467 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8ebe47e363ce4a6c8129169bc6777e63
I20250625 13:52:04.482857 591 catalog_manager.cc:6652] ReplaceTablet: tablet 8ebe47e363ce4a6c8129169bc6777e63 deleted and replaced by tablet 6e2d109e6fce407dad72a2469479b5ed
I20250625 13:52:04.485625 1165 ts_tablet_manager.cc:1905] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.486617 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 067272869c5547648c90f83a34a036aa with delete_type TABLET_DATA_DELETED (replaced by tablet 6340de862fd04c12bb7f85557cfe53d2) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.486618 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ebe47e363ce4a6c8129169bc6777e63 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e2d109e6fce407dad72a2469479b5ed) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.488740 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 067272869c5547648c90f83a34a036aa with delete_type TABLET_DATA_DELETED (replaced by tablet 6340de862fd04c12bb7f85557cfe53d2) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.489336 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ebe47e363ce4a6c8129169bc6777e63 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e2d109e6fce407dad72a2469479b5ed) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:04.498287 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet fd4142572b344456a0d8248be914ad8f (table default.Table_3fe8ea719ca94aaca7b19fadc398d91d [id=33400d976688476eb4450525402f12be], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd4142572b344456a0d8248be914ad8f consensus state has no leader
I20250625 13:52:04.498580 1170 raft_consensus.cc:738] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.500597 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 9a6bc49db6e741c2aa7762b202b3e1f5 (DEFAULT_TABLE table=default.Table_272f40fffdd4480784ee11e2b9fabc45 [id=62898d4eb2a84aa4975a78a7218a6a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.501080 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a6bc49db6e741c2aa7762b202b3e1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.507344 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_81d972e68ae9440f98ef38345914dc4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.507992 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81d972e68ae9440f98ef38345914dc4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.508322 1166 tablet_bootstrap.cc:492] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.508450 1166 ts_tablet_manager.cc:1397] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.275s user 0.016s sys 0.017s
I20250625 13:52:04.509334 1166 raft_consensus.cc:357] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.509544 1166 raft_consensus.cc:383] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.509598 1166 raft_consensus.cc:738] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.510280 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f196bc5d679b4a68a7ab1276485e968c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.510800 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f196bc5d679b4a68a7ab1276485e968c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.518275 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e144ac4cd5e3415092856ff8e91016b3 (DEFAULT_TABLE table=default.Table_8082f0bf0ab7432da22e85d6b82c51f6 [id=7965625755b945fa9054b9cfa081aede]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.518949 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_65b9f8c4792145df86eba8b5f47bf1a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.519496 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65b9f8c4792145df86eba8b5f47bf1a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.454547 1837 raft_consensus.cc:491] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.448614 888 tablet_service.cc:1515] Processing DeleteTablet for tablet d7660a2b70744bd589220b28e6edbc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.522751 1015 tablet_service.cc:1468] Processing CreateTablet for tablet c4db530b54f04fedb18bf993dc60d8da (DEFAULT_TABLE table=default.Table_88d04eecb5a14feda8d8662393261503 [id=ec3dfab61fff4a9ebb227df0e8816a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.523355 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 067272869c5547648c90f83a34a036aa with delete_type TABLET_DATA_DELETED (replaced by tablet 6340de862fd04c12bb7f85557cfe53d2) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.523401 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46316eb725bc47e695af476c0b185504" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.527346 1170 consensus_queue.cc:260] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.528024 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 46316eb725bc47e695af476c0b185504 with delete_type TABLET_DATA_DELETED (replaced by tablet b7d888e628a94f939ecf2571e60b1948) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:04.528744 849 leader_election.cc:343] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.530668 1170 ts_tablet_manager.cc:1428] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.143s user 0.000s sys 0.003s
I20250625 13:52:04.530894 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ebe47e363ce4a6c8129169bc6777e63 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e2d109e6fce407dad72a2469479b5ed) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.530980 1170 tablet_bootstrap.cc:492] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.531489 1837 raft_consensus.cc:513] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.532052 1168 raft_consensus.cc:2241] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:04.533075 849 leader_election.cc:343] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.533226 849 leader_election.cc:304] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:04.533283 851 leader_election.cc:343] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d9bb7ebc40a14ecaa0c692a8e1117db3
I20250625 13:52:04.533406 1710 ts_tablet_manager.cc:1905] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.535527 1171 tablet_bootstrap.cc:492] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.535698 1171 ts_tablet_manager.cc:1397] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.009s
I20250625 13:52:04.536175 1510 raft_consensus.cc:2747] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.533066 1168 raft_consensus.cc:2270] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.537261 1837 leader_election.cc:290] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:04.538406 849 leader_election.cc:343] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: d9bb7ebc40a14ecaa0c692a8e1117db3
I20250625 13:52:04.538547 849 leader_election.cc:304] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:04.538740 849 raft_consensus.cc:2684] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:52:04.539140 1168 ts_tablet_manager.cc:1905] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.538553 1171 raft_consensus.cc:357] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.539530 1171 raft_consensus.cc:383] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.539620 1171 raft_consensus.cc:738] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.539857 1171 consensus_queue.cc:260] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.540895 1165 ts_tablet_manager.cc:1918] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.540994 1165 log.cc:1199] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/637847cac1724bfeb629c4d4013396ca
I20250625 13:52:04.541283 1165 ts_tablet_manager.cc:1939] T 637847cac1724bfeb629c4d4013396ca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.542333 1165 tablet_replica.cc:331] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.542743 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 637847cac1724bfeb629c4d4013396ca (table default.Table_295dd4f121234e659090b8e4cf70faee [id=47cd45886fc94cd7ba5cafa60838d601]) successfully deleted
W20250625 13:52:04.543008 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eabd67b9ceb74bd1a5f4f115c7db0e87 already in progress: creating tablet
I20250625 13:52:04.542502 1165 raft_consensus.cc:2241] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.545007 1165 raft_consensus.cc:2270] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.545729 1165 ts_tablet_manager.cc:1905] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.551774 1165 ts_tablet_manager.cc:1918] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.551911 1165 log.cc:1199] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/54fa042947da4e1b89018640b46b271a
I20250625 13:52:04.552199 1165 ts_tablet_manager.cc:1939] T 54fa042947da4e1b89018640b46b271a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.553130 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 54fa042947da4e1b89018640b46b271a (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]) successfully deleted
I20250625 13:52:04.554114 1165 tablet_replica.cc:331] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.554283 1165 raft_consensus.cc:2241] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.554418 1165 raft_consensus.cc:2270] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.555271 1770 raft_consensus.cc:491] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.555423 1770 raft_consensus.cc:513] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.556017 1770 leader_election.cc:290] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.556167 1165 ts_tablet_manager.cc:1905] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.557101 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_865a0d33634943c88b8a9b87a6cae4b4" } modify_external_catalogs: true
I20250625 13:52:04.557320 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_865a0d33634943c88b8a9b87a6cae4b4" } modify_external_catalogs: true
I20250625 13:52:04.558650 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e5291e5468c04390947d9ea217535cf6" } new_table_name: "default.Table_6a4abf185c2b44d3ba20510778eead0a" modify_external_catalogs: true
I20250625 13:52:04.559137 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a31f10c8ca745b7ad635c622a265450" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.560072 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f196bc5d679b4a68a7ab1276485e968c" } modify_external_catalogs: true
W20250625 13:52:04.560143 849 leader_election.cc:343] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.560247 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f196bc5d679b4a68a7ab1276485e968c" } modify_external_catalogs: true
I20250625 13:52:04.561465 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0e7f3b83e2c7403eb46bc273da03cee8" } modify_external_catalogs: true
I20250625 13:52:04.561760 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0e7f3b83e2c7403eb46bc273da03cee8" } modify_external_catalogs: true
W20250625 13:52:04.562230 851 leader_election.cc:343] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9a31f10c8ca745b7ad635c622a265450
I20250625 13:52:04.562352 851 leader_election.cc:304] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.563846 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_25010c89df9e4e128b8e4531a0bda6d3" } modify_external_catalogs: true
I20250625 13:52:04.564021 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_25010c89df9e4e128b8e4531a0bda6d3" } modify_external_catalogs: true
I20250625 13:52:04.566452 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e144ac4cd5e3415092856ff8e91016b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.567379 589 master_service.cc:948] ReplaceTablet: received request to replace tablet cd4f0857f52449e1b26fde09c6c7c9d4 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:04.568286 1842 raft_consensus.cc:2747] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.569023 1171 ts_tablet_manager.cc:1428] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250625 13:52:04.569660 1171 tablet_bootstrap.cc:492] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.571678 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4db530b54f04fedb18bf993dc60d8da. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:04.572067 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 155737a8d5604eb69524c4d6bea61550 consensus state has no leader
I20250625 13:52:04.575393 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7a2ca389ecd148b9be0e90c130fe6431" } modify_external_catalogs: true
I20250625 13:52:04.575603 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7a2ca389ecd148b9be0e90c130fe6431" } modify_external_catalogs: true
I20250625 13:52:04.576848 595 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ec4dd16ee64c461581ec8e13582cfce6
I20250625 13:52:04.578153 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e8d2ab8f91a045a3a39f3d09459f2a28
I20250625 13:52:04.579128 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e8d2ab8f91a045a3a39f3d09459f2a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.584156 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet e8d2ab8f91a045a3a39f3d09459f2a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.585068 1170 tablet_bootstrap.cc:654] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.585392 1821 raft_consensus.cc:491] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.585637 1821 raft_consensus.cc:513] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.586030 884 tablet_service.cc:1515] Processing DeleteTablet for tablet e8d2ab8f91a045a3a39f3d09459f2a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:04.586365 1165 ts_tablet_manager.cc:1918] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.586480 1165 log.cc:1199] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/8e29e7b55570483b80c6e7a1f227fa7b
I20250625 13:52:04.586774 1165 ts_tablet_manager.cc:1939] T 8e29e7b55570483b80c6e7a1f227fa7b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:52:04.587330 985 leader_election.cc:343] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cd4f0857f52449e1b26fde09c6c7c9d4
I20250625 13:52:04.587771 1171 tablet_bootstrap.cc:654] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.588101 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd4f0857f52449e1b26fde09c6c7c9d4" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:04.588295 912 raft_consensus.cc:2466] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:04.588860 983 leader_election.cc:304] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.589159 1836 raft_consensus.cc:2802] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:04.589253 1836 raft_consensus.cc:491] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:04.589375 1836 raft_consensus.cc:3058] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.590585 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 8e29e7b55570483b80c6e7a1f227fa7b (table default.Table_2b3cf346c6724405a0e96eef702fc43e [id=4891101f0201404ab9d5c88c094ce3ff]) successfully deleted
W20250625 13:52:04.591243 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e3b3d5a2ab7042c2a2f18890363e7a9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b3d5a2ab7042c2a2f18890363e7a9b already in progress: creating tablet
W20250625 13:52:04.591828 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d7660a2b70744bd589220b28e6edbc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7660a2b70744bd589220b28e6edbc42 already in progress: creating tablet
W20250625 13:52:04.592342 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c737daaede054d09a0c07564157423a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c737daaede054d09a0c07564157423a9 already in progress: creating tablet
W20250625 13:52:04.592890 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet eafc402d809245d189f3977162506ef8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eafc402d809245d189f3977162506ef8 already in progress: creating tablet
W20250625 13:52:04.593406 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 067272869c5547648c90f83a34a036aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 067272869c5547648c90f83a34a036aa already in progress: creating tablet
W20250625 13:52:04.593912 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8ebe47e363ce4a6c8129169bc6777e63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ebe47e363ce4a6c8129169bc6777e63 already in progress: creating tablet
W20250625 13:52:04.594409 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e8d2ab8f91a045a3a39f3d09459f2a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8d2ab8f91a045a3a39f3d09459f2a28 already in progress: creating tablet
I20250625 13:52:04.596190 1836 raft_consensus.cc:513] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.597129 778 raft_consensus.cc:1273] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer affa1c3035b940e88ca143be7d35840d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250625 13:52:04.597587 985 leader_election.cc:343] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cd4f0857f52449e1b26fde09c6c7c9d4
I20250625 13:52:04.598258 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd4f0857f52449e1b26fde09c6c7c9d4" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:04.598415 912 raft_consensus.cc:3058] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:04.599169 1842 consensus_queue.cc:1035] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:04.599382 1836 leader_election.cc:290] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.599905 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cbdacccee68e4114ab2784c7bda953df" } new_table_name: "default.Table_fc36d41adb784049b176907652f541ee" modify_external_catalogs: true
I20250625 13:52:04.601389 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_65b6f087b85f4a06a9131796bd8a943e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.601573 912 raft_consensus.cc:2466] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:04.601961 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65b6f087b85f4a06a9131796bd8a943e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.602239 983 leader_election.cc:304] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.603466 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_82770759b43b46a0850155827b18c124" } new_table_name: "default.Table_aa21254fb9d2413e9533648cba22c6aa" modify_external_catalogs: true
I20250625 13:52:04.603678 1836 raft_consensus.cc:2802] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:04.604004 1836 raft_consensus.cc:695] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:04.604717 1836 consensus_queue.cc:237] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:04.606149 849 consensus_peers.cc:487] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:04.606297 1821 leader_election.cc:290] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.607482 1710 ts_tablet_manager.cc:1918] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:04.607625 1710 log.cc:1199] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/9ba63184ca2b4badafcf08309ef75164
I20250625 13:52:04.608067 1710 ts_tablet_manager.cc:1939] T 9ba63184ca2b4badafcf08309ef75164 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:04.608377 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 521) took 2840 ms. Trace:
I20250625 13:52:04.608454 1710 rpcz_store.cc:276] 0625 13:52:01.767678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.767718 (+ 40us) service_pool.cc:225] Handling call
0625 13:52:04.041489 (+2273771us) ts_tablet_manager.cc:1145] Deleting tablet 9ba63184ca2b4badafcf08309ef75164
0625 13:52:04.041501 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.379839 (+338338us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x6160000d2454. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf7f000 00007f18fd291093 00007f18ffd6e137 00007f18ffda01c4 00007f18ffd81c20 00007f18ffd819ad 0000000000a7573c 00007f18f05649df 00007f18f0567cab 00007f18f0567b28
0625 13:52:04.580745 (+200906us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.607354 (+ 26609us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.608322 (+ 968us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":776511232,"tablet-delete.queue_time_us":2273615}
I20250625 13:52:04.609652 1837 raft_consensus.cc:491] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.609771 1837 raft_consensus.cc:513] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.610273 1837 leader_election.cc:290] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:04.611406 851 leader_election.cc:343] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 6771d8e180fe48569a0e016db0cc9b51
W20250625 13:52:04.613925 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99fd8aa1ff1443c29b9c734d896a3bf6 consensus state has no leader
I20250625 13:52:04.628931 1837 raft_consensus.cc:491] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.629086 1837 raft_consensus.cc:513] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.629678 1837 leader_election.cc:290] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:04.630177 1166 consensus_queue.cc:260] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:04.630833 851 leader_election.cc:343] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 799f2edd960d45c1ace9195283a042e2
W20250625 13:52:04.633072 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9def3e1e149a4045a1bb2da76ced8eaf consensus state has no leader
I20250625 13:52:04.634519 1166 ts_tablet_manager.cc:1428] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.126s user 0.019s sys 0.000s
I20250625 13:52:04.636004 1166 tablet_bootstrap.cc:492] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:04.636572 849 leader_election.cc:343] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 799f2edd960d45c1ace9195283a042e2
I20250625 13:52:04.636777 849 leader_election.cc:304] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.637046 1842 raft_consensus.cc:2747] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.637440 1166 tablet_bootstrap.cc:654] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.612686 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6771d8e180fe48569a0e016db0cc9b51" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.629968 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet bb1c7a85da844d34be805843bc60d350 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:04.638187 849 leader_election.cc:343] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.638338 849 leader_election.cc:304] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.638736 1842 raft_consensus.cc:2747] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.638965 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 9ba63184ca2b4badafcf08309ef75164 (table default.Table_dbca3ace8835418e8db2cbc1845d9cfb [id=1cd7ba4de230481e8b0c57c1c585fbd2]) successfully deleted
I20250625 13:52:04.639351 1710 tablet_replica.cc:331] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:04.639555 1710 raft_consensus.cc:2241] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.639844 1710 raft_consensus.cc:2270] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.640379 1710 ts_tablet_manager.cc:1905] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.642369 1168 ts_tablet_manager.cc:1918] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.642505 1168 log.cc:1199] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d9bb7ebc40a14ecaa0c692a8e1117db3
I20250625 13:52:04.656255 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_dcb69af0d4fc42a5982d04bc02404a49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.657426 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_997801fa9593458d9642268646093de6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.657831 1170 tablet_bootstrap.cc:492] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:04.657970 1170 ts_tablet_manager.cc:1397] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.127s user 0.004s sys 0.005s
I20250625 13:52:04.658255 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3ade196aa4334cff912b6e96e532483c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.658701 1170 raft_consensus.cc:357] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.658886 1170 raft_consensus.cc:383] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.658955 1170 raft_consensus.cc:738] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.658913 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8fb483ed4e0140d8ba66be83ccbe58b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.645864 1171 tablet_bootstrap.cc:492] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.659588 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4017f3e1f1534574b8241c6d417fec78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.659663 1171 ts_tablet_manager.cc:1397] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.005s
I20250625 13:52:04.660660 1171 raft_consensus.cc:357] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.660816 1171 raft_consensus.cc:383] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.660969 1171 raft_consensus.cc:738] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.661185 1171 consensus_queue.cc:260] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.661856 1171 ts_tablet_manager.cc:1428] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:04.662197 1171 tablet_bootstrap.cc:492] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.664108 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_21b38804564840edbbc09582cfe8f506" } new_table_name: "default.Table_ff804c2097c04317aa185379facaf553" modify_external_catalogs: true
I20250625 13:52:04.649613 1842 raft_consensus.cc:491] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.665159 1842 raft_consensus.cc:513] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:04.668401 851 leader_election.cc:343] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 01ba67b500b646d7aa55315f7ee23ed6
W20250625 13:52:04.668462 849 consensus_peers.cc:487] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93f449c80b604a6498f6fd1d1ba4770a. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:04.669348 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01ba67b500b646d7aa55315f7ee23ed6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:04.669871 849 leader_election.cc:343] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.670120 849 leader_election.cc:304] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.670344 1510 raft_consensus.cc:2747] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.668903 1842 leader_election.cc:290] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:04.670881 1776 raft_consensus.cc:491] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.671020 1776 raft_consensus.cc:513] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.671728 1776 leader_election.cc:290] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:04.672581 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:04.673450 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:04.673918 711 leader_election.cc:343] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:04.674916 711 leader_election.cc:343] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.675033 711 leader_election.cc:304] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:04.675426 1776 raft_consensus.cc:2747] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.677073 1171 tablet_bootstrap.cc:654] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:04.683178 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e2711256c9042ebb651ae0ca3d02c89 consensus state has no leader
I20250625 13:52:04.659674 1168 ts_tablet_manager.cc:1939] T d9bb7ebc40a14ecaa0c692a8e1117db3 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:04.691895 1170 consensus_queue.cc:260] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.692762 1170 ts_tablet_manager.cc:1428] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250625 13:52:04.693081 1170 tablet_bootstrap.cc:492] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.697901 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 306) took 7321 ms. Trace:
I20250625 13:52:04.698498 1168 rpcz_store.cc:276] 0625 13:51:57.376325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.436477 (+ 60152us) service_pool.cc:225] Handling call
0625 13:52:04.457230 (+7020753us) ts_tablet_manager.cc:1145] Deleting tablet d9bb7ebc40a14ecaa0c692a8e1117db3
0625 13:52:04.457238 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.532923 (+ 75685us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x6160001d0e54. stack: 00007f37024dc336 00007f37024dc0f0 00007f37014d23ad 0000000000971004 0000000000970fcd 000000000097099a 00007f370df3c000 00007f370f24e093 00007f3711d2b137 00007f3711d5d1c4 00007f3711d3ec20 00007f3711d3e9ad 0000000000a7573c 00007f37025219df 00007f3702524cab 00007f3702524b28
0625 13:52:04.639127 (+106204us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.642268 (+ 3141us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.697786 (+ 55518us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":171353984,"tablet-delete.queue_time_us":6931676}
I20250625 13:52:04.700181 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet d9bb7ebc40a14ecaa0c692a8e1117db3 (table default.Table_40de6f83d3af49f1a0bc517c4d0d875e [id=6e50ba5e257842e080a70caa1679952d]) successfully deleted
I20250625 13:52:04.701804 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 307) took 7315 ms. Trace:
I20250625 13:52:04.703244 1510 raft_consensus.cc:491] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.703524 1510 raft_consensus.cc:513] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.704164 1510 leader_election.cc:290] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:04.704265 1166 tablet_bootstrap.cc:492] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.704394 1166 ts_tablet_manager.cc:1397] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.011s
I20250625 13:52:04.704725 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19017cc372af43b2b5074a8990375ede" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:04.705149 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fb483ed4e0140d8ba66be83ccbe58b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.705068 1166 raft_consensus.cc:357] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.705237 1166 raft_consensus.cc:383] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.705292 1166 raft_consensus.cc:738] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:04.705369 849 leader_election.cc:343] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.705480 1166 consensus_queue.cc:260] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:04.705569 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_997801fa9593458d9642268646093de6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.705952 1166 ts_tablet_manager.cc:1428] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250625 13:52:04.706229 1166 tablet_bootstrap.cc:492] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:04.706321 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4017f3e1f1534574b8241c6d417fec78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:04.707496 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ade196aa4334cff912b6e96e532483c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.707557 1166 tablet_bootstrap.cc:654] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.703063 1168 rpcz_store.cc:276] 0625 13:51:57.386906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.525657 (+138751us) service_pool.cc:225] Handling call
0625 13:52:04.700364 (+7174707us) ts_tablet_manager.cc:1145] Deleting tablet 9514bbbda0ba4329a18b0a42e635c6e5
0625 13:52:04.700462 (+ 98us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.701786 (+ 1324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7147453}
W20250625 13:52:04.708642 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9514bbbda0ba4329a18b0a42e635c6e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9514bbbda0ba4329a18b0a42e635c6e5 already in progress: creating tablet
W20250625 13:52:04.708863 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcb69af0d4fc42a5982d04bc02404a49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:04.709642 851 leader_election.cc:343] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 19017cc372af43b2b5074a8990375ede
I20250625 13:52:04.710295 1168 tablet_replica.cc:331] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:04.712651 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d1099e25efdb46c084ac0792c48cc54a" } new_table_name: "default.Table_fbe9775e7dbd44ce9fd173057e030bd7" modify_external_catalogs: true
I20250625 13:52:04.710517 851 leader_election.cc:304] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.715736 1510 raft_consensus.cc:2747] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.720942 1168 raft_consensus.cc:2241] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.721123 1168 raft_consensus.cc:2270] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.721652 1168 ts_tablet_manager.cc:1905] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.725281 1710 ts_tablet_manager.cc:1918] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.726161 1710 log.cc:1199] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/85c72f9ae92a4537979fb479eb4467c4
I20250625 13:52:04.726296 596 catalog_manager.cc:5582] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:04.726600 1710 ts_tablet_manager.cc:1939] T 85c72f9ae92a4537979fb479eb4467c4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:04.726820 596 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
I20250625 13:52:04.727669 743 tablet_service.cc:1468] Processing CreateTablet for tablet b0a13c34b24244d58daa0d67589b5100 (DEFAULT_TABLE table=default.Table_e7a56419acce4220aed05859bbdad3bd [id=705b9a0205344285b5d3cf6db0f4e1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.728088 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a13c34b24244d58daa0d67589b5100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.728474 756 tablet_service.cc:1468] Processing CreateTablet for tablet a028df315713443986c4b362ed17fd99 (DEFAULT_TABLE table=default.Table_103d81d6145e465b96d0044d4fd3d4f5 [id=a9ccf25b783645e3a7e0553e3ed4f8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.728785 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a028df315713443986c4b362ed17fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.729099 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 522) took 2954 ms. Trace:
I20250625 13:52:04.729182 1710 rpcz_store.cc:276] 0625 13:52:01.774903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.774951 (+ 48us) service_pool.cc:225] Handling call
0625 13:52:04.639248 (+2864297us) ts_tablet_manager.cc:1145] Deleting tablet 85c72f9ae92a4537979fb479eb4467c4
0625 13:52:04.639258 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.719742 (+ 80484us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.725146 (+ 5404us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.729078 (+ 3932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2864046}
I20250625 13:52:04.729493 1710 tablet_replica.cc:331] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:04.729660 1710 raft_consensus.cc:2241] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.729807 1710 raft_consensus.cc:2270] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.730940 1710 ts_tablet_manager.cc:1905] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.731485 884 tablet_service.cc:1468] Processing CreateTablet for tablet b0a13c34b24244d58daa0d67589b5100 (DEFAULT_TABLE table=default.Table_e7a56419acce4220aed05859bbdad3bd [id=705b9a0205344285b5d3cf6db0f4e1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.731863 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a13c34b24244d58daa0d67589b5100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.735775 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 85c72f9ae92a4537979fb479eb4467c4 (table default.Table_8fc2c990732d4e658580c2bc2cf67f4d [id=75b34bf006e040d18266d0c90986afeb]) successfully deleted
I20250625 13:52:04.743639 1170 tablet_bootstrap.cc:654] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.744109 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b0a13c34b24244d58daa0d67589b5100 (DEFAULT_TABLE table=default.Table_e7a56419acce4220aed05859bbdad3bd [id=705b9a0205344285b5d3cf6db0f4e1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.749550 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a028df315713443986c4b362ed17fd99 (DEFAULT_TABLE table=default.Table_103d81d6145e465b96d0044d4fd3d4f5 [id=a9ccf25b783645e3a7e0553e3ed4f8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.750715 887 tablet_service.cc:1468] Processing CreateTablet for tablet 9df1f195befe4aed999b41f770547fde (DEFAULT_TABLE table=default.Table_736e3a7bd9a44cfda965390534b95893 [id=c2a98882789d48648370e2d4f217a498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.750742 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6c123375ea524461bd718428af7f2173 (DEFAULT_TABLE table=default.Table_d50d71211aba4a888036f693f4c1336e [id=5d24fd3818f048c3a9f22fefea8ebc67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.751085 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c123375ea524461bd718428af7f2173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.751085 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9df1f195befe4aed999b41f770547fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.750965 1168 ts_tablet_manager.cc:1918] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.751789 1168 log.cc:1199] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/5d7704b38144429b9c95675be84a4730
I20250625 13:52:04.753861 1168 ts_tablet_manager.cc:1939] T 5d7704b38144429b9c95675be84a4730 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:04.755054 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a028df315713443986c4b362ed17fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.757402 743 tablet_service.cc:1468] Processing CreateTablet for tablet 9df1f195befe4aed999b41f770547fde (DEFAULT_TABLE table=default.Table_736e3a7bd9a44cfda965390534b95893 [id=c2a98882789d48648370e2d4f217a498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.757738 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9df1f195befe4aed999b41f770547fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.759918 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 9df1f195befe4aed999b41f770547fde (DEFAULT_TABLE table=default.Table_736e3a7bd9a44cfda965390534b95893 [id=c2a98882789d48648370e2d4f217a498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.760285 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9df1f195befe4aed999b41f770547fde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.769311 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 313) took 7281 ms. Trace:
I20250625 13:52:04.769886 1168 rpcz_store.cc:276] 0625 13:51:57.491833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.671005 (+179172us) service_pool.cc:225] Handling call
0625 13:52:04.710239 (+7039234us) ts_tablet_manager.cc:1145] Deleting tablet 5d7704b38144429b9c95675be84a4730
0625 13:52:04.710253 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.735270 (+ 25017us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.750416 (+ 15146us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.769281 (+ 18865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7039031}
I20250625 13:52:04.770725 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 315) took 7244 ms. Trace:
I20250625 13:52:04.770884 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 5d7704b38144429b9c95675be84a4730 (table default.Table_8c4176baf42b43658ec245c90d86cb98 [id=1786625fb4994b51a8b90c53e7e1714c]) successfully deleted
I20250625 13:52:04.771176 1168 rpcz_store.cc:276] 0625 13:51:57.526356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.691146 (+164790us) service_pool.cc:225] Handling call
0625 13:52:04.770585 (+7079439us) ts_tablet_manager.cc:1145] Deleting tablet 4fa801f7d98e4794b6ca439638f29e69
0625 13:52:04.770606 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.770709 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7078620}
W20250625 13:52:04.771835 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4fa801f7d98e4794b6ca439638f29e69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fa801f7d98e4794b6ca439638f29e69 already in progress: creating tablet
I20250625 13:52:04.772177 1168 tablet_replica.cc:331] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:04.772538 1168 raft_consensus.cc:2241] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.772912 1168 raft_consensus.cc:2270] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.773569 1168 ts_tablet_manager.cc:1905] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.780215 1170 tablet_bootstrap.cc:492] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:04.780370 1170 ts_tablet_manager.cc:1397] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.002s
I20250625 13:52:04.781141 1170 raft_consensus.cc:357] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.781324 1170 raft_consensus.cc:383] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.781440 1170 raft_consensus.cc:738] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.781723 1170 consensus_queue.cc:260] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.782562 1170 ts_tablet_manager.cc:1428] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:04.787024 891 tablet_service.cc:1468] Processing CreateTablet for tablet 6c123375ea524461bd718428af7f2173 (DEFAULT_TABLE table=default.Table_d50d71211aba4a888036f693f4c1336e [id=5d24fd3818f048c3a9f22fefea8ebc67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.787997 889 tablet_service.cc:1468] Processing CreateTablet for tablet a1a9a69d26c14a62b26e0e90157782af (DEFAULT_TABLE table=default.Table_2bcf5e3903254063bd1d49f7f9455737 [id=79ca36b222ec4174bfe1fa924cd0359a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.788343 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a9a69d26c14a62b26e0e90157782af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.794348 884 tablet_service.cc:1468] Processing CreateTablet for tablet a028df315713443986c4b362ed17fd99 (DEFAULT_TABLE table=default.Table_103d81d6145e465b96d0044d4fd3d4f5 [id=a9ccf25b783645e3a7e0553e3ed4f8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.794901 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a028df315713443986c4b362ed17fd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.795794 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c123375ea524461bd718428af7f2173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.796353 1710 ts_tablet_manager.cc:1918] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:04.796453 1710 log.cc:1199] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/15e58fbec94e4ff9b7ac9830b576fe7f
I20250625 13:52:04.796813 1710 ts_tablet_manager.cc:1939] T 15e58fbec94e4ff9b7ac9830b576fe7f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:04.798341 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a13c34b24244d58daa0d67589b5100. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.798525 1166 tablet_bootstrap.cc:492] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.798666 1166 ts_tablet_manager.cc:1397] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.001s
I20250625 13:52:04.799384 1166 raft_consensus.cc:357] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.799646 1166 raft_consensus.cc:383] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.799757 1166 raft_consensus.cc:738] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.799736 1014 tablet_service.cc:1468] Processing CreateTablet for tablet a1a9a69d26c14a62b26e0e90157782af (DEFAULT_TABLE table=default.Table_2bcf5e3903254063bd1d49f7f9455737 [id=79ca36b222ec4174bfe1fa924cd0359a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.800025 1166 consensus_queue.cc:260] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.800737 1166 ts_tablet_manager.cc:1428] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:04.801069 1166 tablet_bootstrap.cc:492] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.802487 1166 tablet_bootstrap.cc:654] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.803397 1170 tablet_bootstrap.cc:492] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.807093 756 tablet_service.cc:1468] Processing CreateTablet for tablet a1a9a69d26c14a62b26e0e90157782af (DEFAULT_TABLE table=default.Table_2bcf5e3903254063bd1d49f7f9455737 [id=79ca36b222ec4174bfe1fa924cd0359a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.807564 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a9a69d26c14a62b26e0e90157782af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.808213 1170 tablet_bootstrap.cc:654] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.815632 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a9a69d26c14a62b26e0e90157782af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.815587 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6c123375ea524461bd718428af7f2173 (DEFAULT_TABLE table=default.Table_d50d71211aba4a888036f693f4c1336e [id=5d24fd3818f048c3a9f22fefea8ebc67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:04.815949 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c123375ea524461bd718428af7f2173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:04.819406 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 523) took 2921 ms. Trace:
I20250625 13:52:04.819496 1710 rpcz_store.cc:276] 0625 13:52:01.897767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.897830 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:04.729453 (+2831623us) ts_tablet_manager.cc:1145] Deleting tablet 15e58fbec94e4ff9b7ac9830b576fe7f
0625 13:52:04.729462 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.765647 (+ 36185us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.796253 (+ 30606us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.819386 (+ 23133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2831385}
I20250625 13:52:04.819806 1710 tablet_replica.cc:331] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:04.819984 1710 raft_consensus.cc:2241] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.819957 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2733881e50814fb49b1cb5ee2e42f772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.820158 1710 raft_consensus.cc:2270] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.820469 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 15e58fbec94e4ff9b7ac9830b576fe7f (table default.Table_72b53bc133ea42568572d075e5493c2e [id=df6a0a5985da4e6e82eb4b99149e24ba]) successfully deleted
W20250625 13:52:04.820552 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2733881e50814fb49b1cb5ee2e42f772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.820704 1710 ts_tablet_manager.cc:1905] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.823432 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 643928ca4fc94ce58431a0525f2abb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.834231 1166 tablet_bootstrap.cc:492] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.834486 1166 ts_tablet_manager.cc:1397] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250625 13:52:04.835294 1170 tablet_bootstrap.cc:492] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:04.835492 1170 ts_tablet_manager.cc:1397] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.001s
I20250625 13:52:04.835376 1166 raft_consensus.cc:357] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.835604 1166 raft_consensus.cc:383] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.835708 1166 raft_consensus.cc:738] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.835986 1166 consensus_queue.cc:260] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.836701 1166 ts_tablet_manager.cc:1428] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:04.837030 1166 tablet_bootstrap.cc:492] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.837926 1170 raft_consensus.cc:357] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.838286 1170 raft_consensus.cc:383] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.838392 1170 raft_consensus.cc:738] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.838474 1166 tablet_bootstrap.cc:654] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.838670 1170 consensus_queue.cc:260] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.839123 1170 ts_tablet_manager.cc:1428] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:04.839672 1170 tablet_bootstrap.cc:492] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.841257 1776 raft_consensus.cc:491] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.841377 1776 raft_consensus.cc:513] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.841958 1776 leader_election.cc:290] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.842538 1851 tablet.cc:1722] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:04.842847 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd2120b4b084691b6604305773e74f9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.843243 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd2120b4b084691b6604305773e74f9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:04.844156 711 leader_election.cc:343] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:04.844344 711 leader_election.cc:343] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.844422 711 leader_election.cc:304] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:04.844869 1776 raft_consensus.cc:2747] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.845966 1170 tablet_bootstrap.cc:654] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.862354 1166 tablet_bootstrap.cc:492] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.862618 1166 ts_tablet_manager.cc:1397] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20250625 13:52:04.863277 1166 raft_consensus.cc:357] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.863520 1166 raft_consensus.cc:383] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.863605 1166 raft_consensus.cc:738] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.863870 1166 consensus_queue.cc:260] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.864496 1166 ts_tablet_manager.cc:1428] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:04.865175 1710 ts_tablet_manager.cc:1918] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:04.865566 1166 tablet_bootstrap.cc:492] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.867026 1168 ts_tablet_manager.cc:1918] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.867039 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]). Expected version 1 got 0
I20250625 13:52:04.867230 1166 tablet_bootstrap.cc:654] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.867794 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cd9897451d5b458bb4135bc901f46208"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.868039 1168 log.cc:1199] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/f73f420a4d7a438eadde3d6af64f0a11
W20250625 13:52:04.868398 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd9897451d5b458bb4135bc901f46208 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.868710 1168 ts_tablet_manager.cc:1939] T f73f420a4d7a438eadde3d6af64f0a11 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:04.869416 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 316) took 7342 ms. Trace:
I20250625 13:52:04.869501 1168 rpcz_store.cc:276] 0625 13:51:57.526616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.691987 (+165371us) service_pool.cc:225] Handling call
0625 13:52:04.772132 (+7080145us) ts_tablet_manager.cc:1145] Deleting tablet f73f420a4d7a438eadde3d6af64f0a11
0625 13:52:04.772142 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.777292 (+ 5150us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.866926 (+ 89634us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.869397 (+ 2471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7080021}
I20250625 13:52:04.874339 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 317) took 7347 ms. Trace:
I20250625 13:52:04.874433 1168 rpcz_store.cc:276] 0625 13:51:57.526843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.692118 (+165275us) service_pool.cc:225] Handling call
0625 13:52:04.874206 (+7182088us) ts_tablet_manager.cc:1145] Deleting tablet 70959ec681524518a7ec2c9ed4016ad1
0625 13:52:04.874217 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.874320 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7181366}
I20250625 13:52:04.875646 1710 log.cc:1199] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/c911d9ecbded417daf70e85c7cbcd52e
I20250625 13:52:04.876103 1710 ts_tablet_manager.cc:1939] T c911d9ecbded417daf70e85c7cbcd52e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:04.878249 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 525) took 2951 ms. Trace:
I20250625 13:52:04.882469 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]). Expected version 1 got 0
I20250625 13:52:04.882725 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_2dbd2e7b0a624bcb99f8edc7918b558d [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 2 got 0
I20250625 13:52:04.882947 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:04.883816 1171 tablet_bootstrap.cc:492] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.883983 1171 ts_tablet_manager.cc:1397] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.222s user 0.011s sys 0.018s
I20250625 13:52:04.884702 1171 raft_consensus.cc:357] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.884894 1171 raft_consensus.cc:383] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.884954 1171 raft_consensus.cc:738] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.885228 1171 consensus_queue.cc:260] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:04.886333 597 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 88f72c555d9842f28ca4bf310da9c53c consensus state has no leader
I20250625 13:52:04.887061 743 tablet_service.cc:1515] Processing DeleteTablet for tablet dbd2120b4b084691b6604305773e74f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.887387 1165 tablet_replica.cc:331] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.887612 1165 raft_consensus.cc:2241] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.887826 1165 raft_consensus.cc:2270] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.888897 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.878388 1710 rpcz_store.cc:276] 0625 13:52:01.926453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.927313 (+ 860us) service_pool.cc:225] Handling call
0625 13:52:04.819757 (+2892444us) ts_tablet_manager.cc:1145] Deleting tablet c911d9ecbded417daf70e85c7cbcd52e
0625 13:52:04.819767 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.849792 (+ 30025us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.865033 (+ 15241us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.878233 (+ 13200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2892208}
I20250625 13:52:04.889371 1710 tablet_replica.cc:331] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:04.889585 1710 raft_consensus.cc:2241] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.889772 1710 raft_consensus.cc:2270] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.889786 1171 ts_tablet_manager.cc:1428] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250625 13:52:04.889978 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.890508 1165 ts_tablet_manager.cc:1905] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.890511 1710 ts_tablet_manager.cc:1905] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.893903 1171 tablet_bootstrap.cc:492] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.899493 1166 tablet_bootstrap.cc:492] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.899641 1166 ts_tablet_manager.cc:1397] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.003s
I20250625 13:52:04.899616 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 318) took 7372 ms. Trace:
I20250625 13:52:04.899701 1168 rpcz_store.cc:276] 0625 13:51:57.526989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.692856 (+165867us) service_pool.cc:225] Handling call
0625 13:52:04.899429 (+7206573us) ts_tablet_manager.cc:1145] Deleting tablet 87f5b36ddb474ed6903c14876b236ba8
0625 13:52:04.899438 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.899592 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7206424}
I20250625 13:52:04.900051 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 314) took 7375 ms. Trace:
I20250625 13:52:04.900132 1168 rpcz_store.cc:276] 0625 13:51:57.524269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.671224 (+146955us) service_pool.cc:225] Handling call
0625 13:52:04.899952 (+7228728us) ts_tablet_manager.cc:1145] Deleting tablet 63d01af6b320489fb99c467c11ed1dc0
0625 13:52:04.899961 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.900034 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7201161}
I20250625 13:52:04.900382 1168 tablet_replica.cc:331] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:04.900346 1166 raft_consensus.cc:357] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.900535 1166 raft_consensus.cc:383] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.900597 1166 raft_consensus.cc:738] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.900604 1168 raft_consensus.cc:2241] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:04.900803 1166 consensus_queue.cc:260] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.900934 1168 raft_consensus.cc:2270] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.901422 1166 ts_tablet_manager.cc:1428] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:04.902082 1171 tablet_bootstrap.cc:654] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.903048 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a637d32e43d647768eebfd52dd249a2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:04.903270 1168 ts_tablet_manager.cc:1905] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:04.903829 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a637d32e43d647768eebfd52dd249a2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.904026 1170 tablet_bootstrap.cc:492] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:04.904171 1170 ts_tablet_manager.cc:1397] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.001s
I20250625 13:52:04.904955 1170 raft_consensus.cc:357] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.905139 1170 raft_consensus.cc:383] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.905244 1170 raft_consensus.cc:738] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.905630 1166 tablet_bootstrap.cc:492] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.907150 1166 tablet_bootstrap.cc:654] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.908046 1170 consensus_queue.cc:260] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.909914 1170 ts_tablet_manager.cc:1428] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:04.910157 1170 tablet_bootstrap.cc:492] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.911588 1170 tablet_bootstrap.cc:654] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.905134 1710 ts_tablet_manager.cc:1918] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:04.923885 1710 log.cc:1199] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d0c5bea3b6214fa8af2792c5e9c411d9
I20250625 13:52:04.924281 1710 ts_tablet_manager.cc:1939] T d0c5bea3b6214fa8af2792c5e9c411d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:04.924652 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 524) took 2998 ms. Trace:
I20250625 13:52:04.924727 1710 rpcz_store.cc:276] 0625 13:52:01.926323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.927535 (+ 1212us) service_pool.cc:225] Handling call
0625 13:52:04.889305 (+2961770us) ts_tablet_manager.cc:1145] Deleting tablet d0c5bea3b6214fa8af2792c5e9c411d9
0625 13:52:04.889315 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.902076 (+ 12761us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.905049 (+ 2973us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.924632 (+ 19583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2961515}
I20250625 13:52:04.925071 1710 tablet_replica.cc:331] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:04.925233 1710 raft_consensus.cc:2241] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.925384 1710 raft_consensus.cc:2270] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.928810 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f033bfa4925b4d1c971789eb77efd92c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:04.864080 1852 tablet.cc:1722] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:04.934311 1165 ts_tablet_manager.cc:1918] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.934432 1165 log.cc:1199] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/dbd2120b4b084691b6604305773e74f9
I20250625 13:52:04.934713 1165 ts_tablet_manager.cc:1939] T dbd2120b4b084691b6604305773e74f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.935575 1165 tablet_replica.cc:331] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:04.935719 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet dbd2120b4b084691b6604305773e74f9 (table default.Table_8910e07e05ad4d71bfff7d5a0d6952f4 [id=2dd7357a611c4866913dfd01fedcbe5b]) successfully deleted
I20250625 13:52:04.935770 1165 raft_consensus.cc:2241] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.935894 1165 raft_consensus.cc:2270] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.936312 1168 ts_tablet_manager.cc:1918] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:04.937793 1168 log.cc:1199] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d524089e6ab044ec961cfc9f587e8f86
I20250625 13:52:04.938505 1171 tablet_bootstrap.cc:492] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.938699 1171 ts_tablet_manager.cc:1397] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.002s
I20250625 13:52:04.939491 1171 raft_consensus.cc:357] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.939747 1171 raft_consensus.cc:383] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.939847 1171 raft_consensus.cc:738] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.940100 1170 tablet_bootstrap.cc:492] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:04.940272 1170 ts_tablet_manager.cc:1397] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250625 13:52:04.940130 1171 consensus_queue.cc:260] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.940972 1170 raft_consensus.cc:357] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.941134 1170 raft_consensus.cc:383] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.941182 1170 raft_consensus.cc:738] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.941359 1170 consensus_queue.cc:260] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.940136 1710 ts_tablet_manager.cc:1905] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.942505 1171 ts_tablet_manager.cc:1428] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:04.942808 1171 tablet_bootstrap.cc:492] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.944144 1166 tablet_bootstrap.cc:492] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.944274 1166 ts_tablet_manager.cc:1397] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.004s
I20250625 13:52:04.944610 1168 ts_tablet_manager.cc:1939] T d524089e6ab044ec961cfc9f587e8f86 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:04.944968 1166 raft_consensus.cc:357] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.945117 1166 raft_consensus.cc:383] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.945180 1166 raft_consensus.cc:738] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.945430 1166 consensus_queue.cc:260] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.946113 1166 ts_tablet_manager.cc:1428] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:04.946400 1166 tablet_bootstrap.cc:492] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.947914 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 326) took 7337 ms. Trace:
I20250625 13:52:04.948047 1166 tablet_bootstrap.cc:654] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.948731 1170 ts_tablet_manager.cc:1428] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:52:04.948989 1170 tablet_bootstrap.cc:492] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.952596 1170 tablet_bootstrap.cc:654] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.953164 1168 rpcz_store.cc:276] 0625 13:51:57.610497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.761246 (+150749us) service_pool.cc:225] Handling call
0625 13:52:04.900341 (+7139095us) ts_tablet_manager.cc:1145] Deleting tablet d524089e6ab044ec961cfc9f587e8f86
0625 13:52:04.900349 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.923019 (+ 22670us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.936220 (+ 13201us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.947894 (+ 11674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7136740}
I20250625 13:52:04.953784 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 332) took 7271 ms. Trace:
I20250625 13:52:04.953891 1168 rpcz_store.cc:276] 0625 13:51:57.682155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.812313 (+130158us) service_pool.cc:225] Handling call
0625 13:52:04.953681 (+7141368us) ts_tablet_manager.cc:1145] Deleting tablet ac4d368347444846880954ced9651404
0625 13:52:04.953689 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.953772 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7141096}
I20250625 13:52:04.951179 1171 tablet_bootstrap.cc:654] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.955392 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 333) took 7262 ms. Trace:
I20250625 13:52:04.955510 1168 rpcz_store.cc:276] 0625 13:51:57.693424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.812611 (+119187us) service_pool.cc:225] Handling call
0625 13:52:04.955265 (+7142654us) ts_tablet_manager.cc:1145] Deleting tablet a22ade8cefac4140b08882c311ea5181
0625 13:52:04.955274 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.955377 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7142436}
I20250625 13:52:04.955956 1168 tablet_replica.cc:331] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:04.956353 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 254afffd283449fdaf5e190ba3028f16
I20250625 13:52:04.956665 1168 raft_consensus.cc:2241] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.956861 1168 raft_consensus.cc:2270] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.957108 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_30935873709947f3adc8b29d8ffc698c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:04.957634 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30935873709947f3adc8b29d8ffc698c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:04.958200 1168 ts_tablet_manager.cc:1905] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.948084 1165 ts_tablet_manager.cc:1905] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:04.959429 1170 tablet_bootstrap.cc:492] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:04.960103 1170 ts_tablet_manager.cc:1397] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250625 13:52:04.960408 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8082f0bf0ab7432da22e85d6b82c51f6" } new_table_name: "default.Table_b0b7c8da7ddc4130aa86a948c67edc38" modify_external_catalogs: true
I20250625 13:52:04.962587 594 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]). Expected version 1 got 0
I20250625 13:52:04.965617 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 254afffd283449fdaf5e190ba3028f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.965827 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 with delete_type TABLET_DATA_DELETED (replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:04.963315 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c9810f2e24dc4b31a0a4fdb1af3ef7db" } modify_external_catalogs: true
I20250625 13:52:04.966490 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c9810f2e24dc4b31a0a4fdb1af3ef7db" } modify_external_catalogs: true
W20250625 13:52:04.968653 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 575043022b1b4ca686775b2aae41a037 consensus state has no leader
I20250625 13:52:04.970858 1171 tablet_bootstrap.cc:492] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.971065 1171 ts_tablet_manager.cc:1397] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250625 13:52:04.972023 1171 raft_consensus.cc:357] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.958400 1710 ts_tablet_manager.cc:1918] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:04.972395 1710 log.cc:1199] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/cf1664afbab647b3baeed6391196bed9
I20250625 13:52:04.972421 1171 raft_consensus.cc:383] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.972501 1171 raft_consensus.cc:738] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.972849 1171 consensus_queue.cc:260] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.973412 1710 ts_tablet_manager.cc:1939] T cf1664afbab647b3baeed6391196bed9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:04.974706 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 526) took 3039 ms. Trace:
I20250625 13:52:04.975497 1171 ts_tablet_manager.cc:1428] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:04.975724 1710 rpcz_store.cc:276] 0625 13:52:01.934817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.934962 (+ 145us) service_pool.cc:225] Handling call
0625 13:52:04.925021 (+2990059us) ts_tablet_manager.cc:1145] Deleting tablet cf1664afbab647b3baeed6391196bed9
0625 13:52:04.925030 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.945048 (+ 20018us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.958295 (+ 13247us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.974684 (+ 16389us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2989840}
I20250625 13:52:04.976029 1171 tablet_bootstrap.cc:492] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.976342 1166 tablet_bootstrap.cc:492] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.972767 1170 raft_consensus.cc:357] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.976444 1170 raft_consensus.cc:383] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.976563 1166 ts_tablet_manager.cc:1397] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20250625 13:52:04.976560 1170 raft_consensus.cc:738] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.976763 1170 consensus_queue.cc:260] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.977344 1166 raft_consensus.cc:357] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.977535 1170 ts_tablet_manager.cc:1428] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 13:52:04.977559 1166 raft_consensus.cc:383] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.977623 1166 raft_consensus.cc:738] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.977910 1166 consensus_queue.cc:260] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.977986 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 527) took 3002 ms. Trace:
I20250625 13:52:04.978057 1710 rpcz_store.cc:276] 0625 13:52:01.975157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.975349 (+ 192us) service_pool.cc:225] Handling call
0625 13:52:04.976399 (+3001050us) ts_tablet_manager.cc:1145] Deleting tablet 0e3ef7f5b474466bbc19fd31c6b6d99c
0625 13:52:04.977857 (+ 1458us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.977971 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3000773}
I20250625 13:52:04.978518 1166 ts_tablet_manager.cc:1428] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:04.978866 1166 tablet_bootstrap.cc:492] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.978446 1170 tablet_bootstrap.cc:492] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:04.980507 1166 tablet_bootstrap.cc:654] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.981429 1170 tablet_bootstrap.cc:654] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.981619 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 528) took 2989 ms. Trace:
I20250625 13:52:04.982856 1710 rpcz_store.cc:276] 0625 13:52:01.992632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.992704 (+ 72us) service_pool.cc:225] Handling call
0625 13:52:04.979371 (+2986667us) ts_tablet_manager.cc:1145] Deleting tablet 2404e712e18146b68757bd77362c1358
0625 13:52:04.979380 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.981592 (+ 2212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2986429}
I20250625 13:52:04.983064 1165 ts_tablet_manager.cc:1918] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.983155 1165 log.cc:1199] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/c666ff2d6b9b4ed4a18e370826c143fc
I20250625 13:52:04.983443 1165 ts_tablet_manager.cc:1939] T c666ff2d6b9b4ed4a18e370826c143fc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:04.983500 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 538) took 2889 ms. Trace:
I20250625 13:52:04.983642 1710 rpcz_store.cc:276] 0625 13:52:02.093851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.094002 (+ 151us) service_pool.cc:225] Handling call
0625 13:52:04.983385 (+2889383us) ts_tablet_manager.cc:1145] Deleting tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57
0625 13:52:04.983400 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.983484 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2864224}
I20250625 13:52:04.983853 1171 tablet_bootstrap.cc:654] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:04.984413 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 537) took 2904 ms. Trace:
I20250625 13:52:04.987126 1168 ts_tablet_manager.cc:1918] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:04.987352 1168 log.cc:1199] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/27092991fcad475fabc77d1f8bab0582
I20250625 13:52:04.987730 1168 ts_tablet_manager.cc:1939] T 27092991fcad475fabc77d1f8bab0582 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:04.988422 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 334) took 7290 ms. Trace:
I20250625 13:52:04.991096 1776 raft_consensus.cc:491] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.991238 1776 raft_consensus.cc:513] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.991823 1776 leader_election.cc:290] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:04.992498 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:04.992574 1166 tablet_bootstrap.cc:492] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:04.992683 1166 ts_tablet_manager.cc:1397] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250625 13:52:04.992678 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:04.992223 1168 rpcz_store.cc:276] 0625 13:51:57.698152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.812850 (+114698us) service_pool.cc:225] Handling call
0625 13:52:04.955913 (+7143063us) ts_tablet_manager.cc:1145] Deleting tablet 27092991fcad475fabc77d1f8bab0582
0625 13:52:04.955921 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.966256 (+ 10335us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.987011 (+ 20755us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:04.988403 (+ 1392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7142855}
I20250625 13:52:04.993081 1168 tablet_replica.cc:331] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:04.993556 1168 raft_consensus.cc:2241] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:04.993500 1166 raft_consensus.cc:357] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:04.993553 711 leader_election.cc:343] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.993649 1166 raft_consensus.cc:383] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.993721 1166 raft_consensus.cc:738] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:04.993752 711 leader_election.cc:343] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:04.993872 711 leader_election.cc:304] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:04.994273 1776 raft_consensus.cc:2747] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:04.994617 1166 consensus_queue.cc:260] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.995158 1168 raft_consensus.cc:2270] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:04.995757 1166 ts_tablet_manager.cc:1428] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250625 13:52:04.996104 1171 tablet_bootstrap.cc:492] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:04.996407 1166 tablet_bootstrap.cc:492] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:04.996419 1171 ts_tablet_manager.cc:1397] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250625 13:52:04.996860 1836 raft_consensus.cc:491] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:04.996999 1836 raft_consensus.cc:513] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:04.997318 1171 raft_consensus.cc:357] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.997486 1171 raft_consensus.cc:383] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:04.997543 1171 raft_consensus.cc:738] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:04.997635 1836 leader_election.cc:290] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:04.997766 1171 consensus_queue.cc:260] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:04.998124 1166 tablet_bootstrap.cc:654] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:04.998298 985 leader_election.cc:343] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 991bb428def545fd979e9f1a80ce4d18
I20250625 13:52:04.998646 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "991bb428def545fd979e9f1a80ce4d18" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:04.998585 1710 rpcz_store.cc:276] 0625 13:52:02.080311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.080460 (+ 149us) service_pool.cc:225] Handling call
0625 13:52:04.984319 (+2903859us) ts_tablet_manager.cc:1145] Deleting tablet 553af7c7326240838f37e3861d4e3357
0625 13:52:04.984328 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:04.984399 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2864163}
I20250625 13:52:04.998814 912 raft_consensus.cc:2466] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:04.999122 1171 ts_tablet_manager.cc:1428] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:04.999433 1171 tablet_bootstrap.cc:492] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:04.999541 983 leader_election.cc:304] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:04.999830 1836 raft_consensus.cc:2802] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:04.999915 1836 raft_consensus.cc:491] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:04.999979 1836 raft_consensus.cc:3058] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.001011 1171 tablet_bootstrap.cc:654] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.002213 1168 ts_tablet_manager.cc:1905] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.002655 1836 raft_consensus.cc:513] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.003508 1836 leader_election.cc:290] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:05.004150 985 leader_election.cc:343] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 991bb428def545fd979e9f1a80ce4d18
I20250625 13:52:05.004258 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "991bb428def545fd979e9f1a80ce4d18" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:05.004446 912 raft_consensus.cc:3058] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.004945 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 536) took 2931 ms. Trace:
I20250625 13:52:05.005023 1710 rpcz_store.cc:276] 0625 13:52:02.073717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.075788 (+ 2071us) service_pool.cc:225] Handling call
0625 13:52:05.004817 (+2929029us) ts_tablet_manager.cc:1145] Deleting tablet 48645c913fc24c03a922cf6ea7531022
0625 13:52:05.004829 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.004924 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2881850}
I20250625 13:52:05.006476 1836 raft_consensus.cc:491] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.006577 1836 raft_consensus.cc:513] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.006978 912 raft_consensus.cc:2466] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:05.007225 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1a6d0dbbe2894e61b0ea08f98bce61e7
I20250625 13:52:05.007263 1510 raft_consensus.cc:491] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.007385 1510 raft_consensus.cc:513] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.007941 1510 leader_election.cc:290] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.008414 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.008530 1836 leader_election.cc:290] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.008646 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ab663b7eba452c9f9775c0f568010b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.009128 849 leader_election.cc:343] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:05.009912 851 leader_election.cc:343] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 34ab663b7eba452c9f9775c0f568010b
I20250625 13:52:05.010018 851 leader_election.cc:304] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.010505 1510 raft_consensus.cc:2747] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.009123 983 leader_election.cc:304] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.010943 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b7d888e628a94f939ecf2571e60b1948
I20250625 13:52:05.011104 1836 raft_consensus.cc:2802] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:05.012508 596 master_service.cc:948] ReplaceTablet: received request to replace tablet b1acc41a93f6433588727811903921fe from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.013119 743 tablet_service.cc:1515] Processing DeleteTablet for tablet b7d888e628a94f939ecf2571e60b1948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.014276 985 leader_election.cc:343] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: bb1c7a85da844d34be805843bc60d350
I20250625 13:52:05.020987 1171 tablet_bootstrap.cc:492] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.021183 1171 ts_tablet_manager.cc:1397] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250625 13:52:05.021920 1171 raft_consensus.cc:357] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.022150 1171 raft_consensus.cc:383] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.022248 1171 raft_consensus.cc:738] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.022490 1171 consensus_queue.cc:260] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.023332 1836 raft_consensus.cc:695] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:05.023010 1171 ts_tablet_manager.cc:1428] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:05.023782 1171 tablet_bootstrap.cc:492] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.023715 1836 consensus_queue.cc:237] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.024111 1166 tablet_bootstrap.cc:492] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.024276 1166 ts_tablet_manager.cc:1397] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20250625 13:52:05.024708 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet c666ff2d6b9b4ed4a18e370826c143fc (table default.Table_badc8f5a2d5040af96b9e7c9758c6417 [id=8a37f6c60b414741a615d1fa7f9a4f3c]) successfully deleted
W20250625 13:52:05.024730 983 leader_election.cc:343] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: bb1c7a85da844d34be805843bc60d350
I20250625 13:52:05.024827 983 leader_election.cc:304] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:05.025063 1821 raft_consensus.cc:2747] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.025103 1166 raft_consensus.cc:357] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.025303 1166 raft_consensus.cc:383] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.025398 1166 raft_consensus.cc:738] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.025640 1166 consensus_queue.cc:260] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.026371 1171 tablet_bootstrap.cc:654] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.026688 1166 ts_tablet_manager.cc:1428] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:05.027017 1166 tablet_bootstrap.cc:492] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.027803 1165 tablet_replica.cc:331] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.028996 1166 tablet_bootstrap.cc:654] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:05.031649 581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:33015, user_credentials={real_user=slave}} blocked reactor thread for 103024us
I20250625 13:52:05.032995 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet c911d9ecbded417daf70e85c7cbcd52e (table default.Table_a90cc30efd7145fdbaa7ebdc5db04107 [id=dec0cb5442804b6398a22dbd8c9b0f3a]) successfully deleted
I20250625 13:52:05.033000 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 055f88e118454e32baa578cae9aceb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.033475 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet d0c5bea3b6214fa8af2792c5e9c411d9 (table default.Table_77082a1ec23a48adb1f251559dc90002 [id=d482f250757e4eaaa5a5a4daa115bd34]) successfully deleted
I20250625 13:52:05.033742 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet cf1664afbab647b3baeed6391196bed9 (table default.Table_98ae9c7310a24e8d988b919a5ec496e7 [id=05b9dac188b34d0b8022defddccbff26]) successfully deleted
I20250625 13:52:05.030586 1168 ts_tablet_manager.cc:1918] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.034323 1168 log.cc:1199] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/9ba63184ca2b4badafcf08309ef75164
W20250625 13:52:05.034379 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e3ef7f5b474466bbc19fd31c6b6d99c already in progress: creating tablet
W20250625 13:52:05.036403 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2404e712e18146b68757bd77362c1358 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2404e712e18146b68757bd77362c1358 already in progress: creating tablet
W20250625 13:52:05.036720 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 already in progress: creating tablet
W20250625 13:52:05.036959 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 553af7c7326240838f37e3861d4e3357 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 553af7c7326240838f37e3861d4e3357 already in progress: creating tablet
W20250625 13:52:05.037205 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 48645c913fc24c03a922cf6ea7531022 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48645c913fc24c03a922cf6ea7531022 already in progress: creating tablet
I20250625 13:52:05.037837 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet f73f420a4d7a438eadde3d6af64f0a11 (table default.Table_9e86330e77034242a9aa4fbd274c3151 [id=0b06b619484e459f949d8acb2bb19512]) successfully deleted
I20250625 13:52:05.039831 1168 ts_tablet_manager.cc:1939] T 9ba63184ca2b4badafcf08309ef75164 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:05.039829 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 70959ec681524518a7ec2c9ed4016ad1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70959ec681524518a7ec2c9ed4016ad1 already in progress: creating tablet
W20250625 13:52:05.040189 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 63d01af6b320489fb99c467c11ed1dc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63d01af6b320489fb99c467c11ed1dc0 already in progress: creating tablet
I20250625 13:52:05.040714 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 335) took 7338 ms. Trace:
I20250625 13:52:05.040853 1168 rpcz_store.cc:276] 0625 13:51:57.702174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.813074 (+110900us) service_pool.cc:225] Handling call
0625 13:52:04.993033 (+7179959us) ts_tablet_manager.cc:1145] Deleting tablet 9ba63184ca2b4badafcf08309ef75164
0625 13:52:04.993043 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.017295 (+ 24252us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.030486 (+ 13191us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.040693 (+ 10207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7179737}
I20250625 13:52:05.041451 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 336) took 7329 ms. Trace:
I20250625 13:52:05.041484 1170 tablet_bootstrap.cc:492] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:05.041637 1170 ts_tablet_manager.cc:1397] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.002s
I20250625 13:52:05.041611 1168 rpcz_store.cc:276] 0625 13:51:57.711532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.845373 (+133841us) service_pool.cc:225] Handling call
0625 13:52:05.041328 (+7195955us) ts_tablet_manager.cc:1145] Deleting tablet a0e81d4c904248eba858462bcf31c5d9
0625 13:52:05.041338 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.041435 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7195719}
I20250625 13:52:05.042048 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet d524089e6ab044ec961cfc9f587e8f86 (table default.Table_888aed1788b245e5b873ac73a0dcfba5 [id=c22a16ca0ca649238ac510cb5542bd77]) successfully deleted
I20250625 13:52:05.042330 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 339) took 7329 ms. Trace:
W20250625 13:52:05.042331 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ac4d368347444846880954ced9651404 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac4d368347444846880954ced9651404 already in progress: creating tablet
I20250625 13:52:05.042697 1168 rpcz_store.cc:276] 0625 13:51:57.712398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.861690 (+149292us) service_pool.cc:225] Handling call
0625 13:52:05.042217 (+7180527us) ts_tablet_manager.cc:1145] Deleting tablet 1effed0fa10d409abcb31dbba54861d6
0625 13:52:05.042227 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.042316 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7180297}
W20250625 13:52:05.043102 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a22ade8cefac4140b08882c311ea5181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a22ade8cefac4140b08882c311ea5181 already in progress: creating tablet
I20250625 13:52:05.031311 1165 raft_consensus.cc:2241] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.006309 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 540) took 2879 ms. Trace:
I20250625 13:52:05.043438 1710 rpcz_store.cc:276] 0625 13:52:02.127447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.128381 (+ 934us) service_pool.cc:225] Handling call
0625 13:52:05.006187 (+2877806us) ts_tablet_manager.cc:1145] Deleting tablet 9c1deb52366c404cbdc08076d4bcb3d1
0625 13:52:05.006199 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.006294 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871358}
I20250625 13:52:05.043597 1165 raft_consensus.cc:2270] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.044126 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 546) took 2888 ms. Trace:
I20250625 13:52:05.044171 1166 tablet_bootstrap.cc:492] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.044286 1166 ts_tablet_manager.cc:1397] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250625 13:52:05.044395 1165 ts_tablet_manager.cc:1905] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.043280 1168 tablet_replica.cc:331] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.045277 1168 raft_consensus.cc:2241] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.046749 1171 tablet_bootstrap.cc:492] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.046878 1171 ts_tablet_manager.cc:1397] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250625 13:52:05.047746 1171 raft_consensus.cc:357] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.047914 1171 raft_consensus.cc:383] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.047986 1171 raft_consensus.cc:738] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.048204 1171 consensus_queue.cc:260] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.048660 1171 ts_tablet_manager.cc:1428] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:05.048940 1171 tablet_bootstrap.cc:492] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.050487 1171 tablet_bootstrap.cc:654] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.051314 1168 raft_consensus.cc:2270] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.051894 1168 ts_tablet_manager.cc:1905] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.051887 1166 raft_consensus.cc:357] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.052062 1166 raft_consensus.cc:383] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.052129 1166 raft_consensus.cc:738] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.052332 1166 consensus_queue.cc:260] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.052829 1166 ts_tablet_manager.cc:1428] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250625 13:52:05.053107 1166 tablet_bootstrap.cc:492] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.054749 1166 tablet_bootstrap.cc:654] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.044273 1710 rpcz_store.cc:276] 0625 13:52:02.155333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.155388 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:05.044012 (+2888624us) ts_tablet_manager.cc:1145] Deleting tablet fb4973cbffc743dba98d7b7c9f76be3f
0625 13:52:05.044021 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.044110 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2887037}
I20250625 13:52:05.056907 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 562) took 2664 ms. Trace:
I20250625 13:52:05.056995 1710 rpcz_store.cc:276] 0625 13:52:02.395813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.397613 (+ 1800us) service_pool.cc:225] Handling call
0625 13:52:05.056758 (+2659145us) ts_tablet_manager.cc:1145] Deleting tablet 69672579a83c41d09ee7c78a6fa7d746
0625 13:52:05.056769 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.056889 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2634170}
I20250625 13:52:05.057314 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 565) took 2640 ms. Trace:
I20250625 13:52:05.057389 1710 rpcz_store.cc:276] 0625 13:52:02.416490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.419726 (+ 3236us) service_pool.cc:225] Handling call
0625 13:52:05.057226 (+2637500us) ts_tablet_manager.cc:1145] Deleting tablet 5d4fa8abea8d4eee8651be9bf4b2b34f
0625 13:52:05.057236 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.057303 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620516}
I20250625 13:52:05.057682 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 564) took 2641 ms. Trace:
I20250625 13:52:05.057755 1710 rpcz_store.cc:276] 0625 13:52:02.416325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.419828 (+ 3503us) service_pool.cc:225] Handling call
0625 13:52:05.057587 (+2637759us) ts_tablet_manager.cc:1145] Deleting tablet 8063122ea7b24b6790f5c3cc884e09be
0625 13:52:05.057595 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.057669 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2522895}
I20250625 13:52:05.057994 1710 tablet_replica.cc:331] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.058161 1710 raft_consensus.cc:2241] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:05.047377 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e144ac4cd5e3415092856ff8e91016b3 consensus state has no leader
I20250625 13:52:05.061187 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_88d04eecb5a14feda8d8662393261503" } new_table_name: "default.Table_57e8a917a4764270a0b6ef4a4cd53dc8" modify_external_catalogs: true
I20250625 13:52:05.051764 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_67a5cc5e1f4449029550030a01425b01" } new_table_name: "default.Table_722a6167ceef49f3b3c899c6fae0f44a" modify_external_catalogs: true
I20250625 13:52:05.052253 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_70a5975b997b4664ae947803f73e1a27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.063892 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70a5975b997b4664ae947803f73e1a27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.064433 1168 ts_tablet_manager.cc:1918] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.064546 1168 log.cc:1199] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/02f2a2b595bc43309a29b3f1bc5d3766
I20250625 13:52:05.052682 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ceec0bc74ee04dd5b21dcb6735342a42" } new_table_name: "default.Table_4ce1c9ba4cf14145821fc2430363bf62" modify_external_catalogs: true
I20250625 13:52:05.064729 1165 ts_tablet_manager.cc:1918] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.064893 1168 ts_tablet_manager.cc:1939] T 02f2a2b595bc43309a29b3f1bc5d3766 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.064909 1165 log.cc:1199] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/58aa3c21191e4bea9c5395d6e32b8717
I20250625 13:52:05.065356 1165 ts_tablet_manager.cc:1939] T 58aa3c21191e4bea9c5395d6e32b8717 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.052997 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dc86aa488ba944c689ae858bc3800718" } modify_external_catalogs: true
I20250625 13:52:05.065672 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 340) took 7353 ms. Trace:
I20250625 13:52:05.065694 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dc86aa488ba944c689ae858bc3800718" } modify_external_catalogs: true
I20250625 13:52:05.065778 1168 rpcz_store.cc:276] 0625 13:51:57.712474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:57.861939 (+149465us) service_pool.cc:225] Handling call
0625 13:52:05.043238 (+7181299us) ts_tablet_manager.cc:1145] Deleting tablet 02f2a2b595bc43309a29b3f1bc5d3766
0625 13:52:05.043247 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.060001 (+ 16754us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.064330 (+ 4329us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.065651 (+ 1321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7181139}
I20250625 13:52:05.066056 1170 raft_consensus.cc:357] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.066262 1170 raft_consensus.cc:383] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.066367 1170 raft_consensus.cc:738] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.066648 1170 consensus_queue.cc:260] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.066895 1710 raft_consensus.cc:2270] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.068943 1710 ts_tablet_manager.cc:1905] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.070483 1510 raft_consensus.cc:491] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.070609 1510 raft_consensus.cc:513] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.071219 1510 leader_election.cc:290] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.071841 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7c3f0ca6fa5482db7f57143911c838f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:05.071995 1046 raft_consensus.cc:2466] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:05.072299 1171 tablet_bootstrap.cc:492] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.072418 1171 ts_tablet_manager.cc:1397] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250625 13:52:05.072777 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7c3f0ca6fa5482db7f57143911c838f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:05.072937 774 raft_consensus.cc:2466] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:05.073493 849 leader_election.cc:304] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:05.073720 1837 raft_consensus.cc:491] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.069308 1170 ts_tablet_manager.cc:1428] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250625 13:52:05.073815 1837 raft_consensus.cc:513] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.074105 1171 raft_consensus.cc:357] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.074231 1171 raft_consensus.cc:383] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.074277 1171 raft_consensus.cc:738] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.074339 1837 leader_election.cc:290] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.074435 1837 raft_consensus.cc:2802] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:05.074496 1837 raft_consensus.cc:491] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:05.074548 1837 raft_consensus.cc:3058] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.074553 1170 tablet_bootstrap.cc:492] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:05.075099 1165 tablet_replica.cc:331] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.075258 1165 raft_consensus.cc:2241] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.075420 1165 raft_consensus.cc:2270] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.076108 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 58aa3c21191e4bea9c5395d6e32b8717 (table default.Table_759e39cb7c154d6281c46789f97f0691 [id=5efdc94a9cf74a3daa545d4d93684eaa]) successfully deleted
I20250625 13:52:05.076272 1170 tablet_bootstrap.cc:654] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:05.076397 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 254afffd283449fdaf5e190ba3028f16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 254afffd283449fdaf5e190ba3028f16 already in progress: creating tablet
I20250625 13:52:05.076541 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:05.076731 774 raft_consensus.cc:2466] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:05.077313 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:05.077852 851 leader_election.cc:304] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:05.078181 849 leader_election.cc:343] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.078424 1748 raft_consensus.cc:491] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.078447 1842 raft_consensus.cc:2802] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:05.078522 1842 raft_consensus.cc:491] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:05.078573 1842 raft_consensus.cc:3058] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.078510 1748 raft_consensus.cc:513] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.079038 1748 leader_election.cc:290] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:05.080555 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54fa042947da4e1b89018640b46b271a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.081027 851 leader_election.cc:343] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 54fa042947da4e1b89018640b46b271a
W20250625 13:52:05.081254 849 leader_election.cc:343] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.081337 849 leader_election.cc:304] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.081728 1510 raft_consensus.cc:2747] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.081832 1842 raft_consensus.cc:513] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.082777 1842 leader_election.cc:290] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.083377 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:05.083557 774 raft_consensus.cc:3058] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.083763 1837 raft_consensus.cc:513] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.084040 1165 ts_tablet_manager.cc:1905] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.084235 1837 leader_election.cc:290] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.084825 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7c3f0ca6fa5482db7f57143911c838f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:05.084972 777 raft_consensus.cc:3058] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.086283 774 raft_consensus.cc:2466] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:05.087417 777 raft_consensus.cc:2466] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:05.087916 1710 ts_tablet_manager.cc:1918] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250625 13:52:05.087970 851 leader_election.cc:304] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:05.088089 1710 log.cc:1199] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/6e0f2fa5d5614ad394e6e12ec62814d4
I20250625 13:52:05.088228 851 leader_election.cc:304] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:05.088307 1166 tablet_bootstrap.cc:492] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.088420 1166 ts_tablet_manager.cc:1397] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250625 13:52:05.088424 1710 ts_tablet_manager.cc:1939] T 6e0f2fa5d5614ad394e6e12ec62814d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.088531 1837 raft_consensus.cc:2802] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:05.088510 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7c3f0ca6fa5482db7f57143911c838f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:05.088654 1045 raft_consensus.cc:3058] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.088759 1510 raft_consensus.cc:2802] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:05.088847 1510 raft_consensus.cc:695] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:05.089049 1510 consensus_queue.cc:237] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.089176 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 561) took 2712 ms. Trace:
I20250625 13:52:05.089187 1166 raft_consensus.cc:357] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.089311 1166 raft_consensus.cc:383] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.089359 1166 raft_consensus.cc:738] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.089250 1710 rpcz_store.cc:276] 0625 13:52:02.376312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.397123 (+ 20811us) service_pool.cc:225] Handling call
0625 13:52:05.057958 (+2660835us) ts_tablet_manager.cc:1145] Deleting tablet 6e0f2fa5d5614ad394e6e12ec62814d4
0625 13:52:05.057965 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.072338 (+ 14373us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.087805 (+ 15467us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.089156 (+ 1351us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":686,"tablet-delete.queue_time_us":2460583}
I20250625 13:52:05.089555 1166 consensus_queue.cc:260] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.089892 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 569) took 2436 ms. Trace:
I20250625 13:52:05.089980 1710 rpcz_store.cc:276] 0625 13:52:02.653519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.653944 (+ 425us) service_pool.cc:225] Handling call
0625 13:52:05.089777 (+2435833us) ts_tablet_manager.cc:1145] Deleting tablet 57afeff143c846289c6159974b00d7ba
0625 13:52:05.089787 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.089876 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2410779}
I20250625 13:52:05.090124 1166 ts_tablet_manager.cc:1428] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:05.090389 1837 raft_consensus.cc:695] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:05.090610 1837 consensus_queue.cc:237] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.091647 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 566) took 2559 ms. Trace:
I20250625 13:52:05.091701 1166 tablet_bootstrap.cc:492] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.091987 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:05.091986 1710 rpcz_store.cc:276] 0625 13:52:02.532318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.534997 (+ 2679us) service_pool.cc:225] Handling call
0625 13:52:05.091540 (+2556543us) ts_tablet_manager.cc:1145] Deleting tablet a6ff3b675aba42adb035ff5d1404a889
0625 13:52:05.091550 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.091632 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406917}
I20250625 13:52:05.092705 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 572) took 2420 ms. Trace:
W20250625 13:52:05.092831 849 leader_election.cc:343] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.093106 1710 rpcz_store.cc:276] 0625 13:52:02.680425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.681369 (+ 944us) service_pool.cc:225] Handling call
0625 13:52:05.092585 (+2411216us) ts_tablet_manager.cc:1145] Deleting tablet 60ef50a3624b4c79acc803ff0e2ed364
0625 13:52:05.092593 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.092691 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407354}
W20250625 13:52:05.093498 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4db530b54f04fedb18bf993dc60d8da consensus state has no leader
I20250625 13:52:05.093796 1166 tablet_bootstrap.cc:654] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.093842 593 master_service.cc:948] ReplaceTablet: received request to replace tablet c4db530b54f04fedb18bf993dc60d8da from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.094331 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 571) took 2433 ms. Trace:
W20250625 13:52:05.094902 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 156da2a965c64d1090a4f9d4eafdec70 consensus state has no leader
I20250625 13:52:05.095140 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 58a532e889244b088562081e444234a3 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.074442 1171 consensus_queue.cc:260] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.096437 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 0b129117db064a15b27e82439637cf22 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.096897 595 master_service.cc:948] ReplaceTablet: received request to replace tablet dbd2120b4b084691b6604305773e74f9 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.097249 1171 ts_tablet_manager.cc:1428] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250625 13:52:05.097530 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0949521f201f43578a2039b556e84501"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.098249 1171 tablet_bootstrap.cc:492] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.095992 1045 raft_consensus.cc:2466] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:05.098724 1710 rpcz_store.cc:276] 0625 13:52:02.660462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.679025 (+ 18563us) service_pool.cc:225] Handling call
0625 13:52:05.094014 (+2414989us) ts_tablet_manager.cc:1145] Deleting tablet 39540b81fd4746858e65be49ded0e1df
0625 13:52:05.094022 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.094315 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2401723}
W20250625 13:52:05.098908 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0949521f201f43578a2039b556e84501 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.099522 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 570) took 2443 ms. Trace:
I20250625 13:52:05.099857 1710 rpcz_store.cc:276] 0625 13:52:02.655599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.672866 (+ 17267us) service_pool.cc:225] Handling call
0625 13:52:05.099285 (+2426419us) ts_tablet_manager.cc:1145] Deleting tablet 2c8cf5d6cd624a1797486c69bd2098fd
0625 13:52:05.099293 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.099505 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2382465}
I20250625 13:52:05.100538 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 601) took 2097 ms. Trace:
I20250625 13:52:05.100961 1710 rpcz_store.cc:276] 0625 13:52:03.002757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.344137 (+341380us) service_pool.cc:225] Handling call
0625 13:52:05.100420 (+1756283us) ts_tablet_manager.cc:1145] Deleting tablet d2f545a5934748b2af767d37ea937fca
0625 13:52:05.100429 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.100502 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742863}
I20250625 13:52:05.101431 1168 tablet_replica.cc:331] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.101615 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 612) took 1683 ms. Trace:
I20250625 13:52:05.101989 1710 rpcz_store.cc:276] 0625 13:52:03.417721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.418085 (+ 364us) service_pool.cc:225] Handling call
0625 13:52:05.101511 (+1683426us) ts_tablet_manager.cc:1145] Deleting tablet 3016763b9e294b9da2861fe97e57e241
0625 13:52:05.101520 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.101590 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673021}
I20250625 13:52:05.102624 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 613) took 1684 ms. Trace:
I20250625 13:52:05.102789 1165 ts_tablet_manager.cc:1918] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.102845 1168 raft_consensus.cc:2241] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.102959 1710 rpcz_store.cc:276] 0625 13:52:03.417886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.418035 (+ 149us) service_pool.cc:225] Handling call
0625 13:52:05.102524 (+1684489us) ts_tablet_manager.cc:1145] Deleting tablet 93fdde7d068b400ba7070166eff63502
0625 13:52:05.102532 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.102596 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666084}
I20250625 13:52:05.103430 1168 raft_consensus.cc:2270] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.103631 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 617) took 1681 ms. Trace:
I20250625 13:52:05.103966 1710 rpcz_store.cc:276] 0625 13:52:03.422252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.477194 (+ 54942us) service_pool.cc:225] Handling call
0625 13:52:05.103534 (+1626340us) ts_tablet_manager.cc:1145] Deleting tablet f197ea67c34546a7981bcd9df94702e6
0625 13:52:05.103542 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.103618 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1623331}
I20250625 13:52:05.104262 1168 ts_tablet_manager.cc:1905] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.104494 1165 log.cc:1199] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/11711546f0e14f9a9eabf003f49a69b7
I20250625 13:52:05.104727 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 618) took 1682 ms. Trace:
I20250625 13:52:05.105069 1165 ts_tablet_manager.cc:1939] T 11711546f0e14f9a9eabf003f49a69b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.105064 1710 rpcz_store.cc:276] 0625 13:52:03.422343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.480230 (+ 57887us) service_pool.cc:225] Handling call
0625 13:52:05.104639 (+1624409us) ts_tablet_manager.cc:1145] Deleting tablet c0f461a7a7ef45009279716a3024ee26
0625 13:52:05.104646 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.104712 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1622625}
I20250625 13:52:05.105729 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 619) took 1683 ms. Trace:
I20250625 13:52:05.106070 1710 rpcz_store.cc:276] 0625 13:52:03.422423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.482043 (+ 59620us) service_pool.cc:225] Handling call
0625 13:52:05.105636 (+1623593us) ts_tablet_manager.cc:1145] Deleting tablet 0e77736191544e8da32ca9927f0a5804
0625 13:52:05.105646 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.105715 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1623428}
I20250625 13:52:05.106683 1710 tablet_replica.cc:331] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.107064 1710 raft_consensus.cc:2241] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.107539 1710 pending_rounds.cc:70] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:05.102468 1171 tablet_bootstrap.cc:654] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.111912 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_2dbd2e7b0a624bcb99f8edc7918b558d [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 2 got 0
I20250625 13:52:05.113328 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_931f25deb4904d2fa74ab9437e733369"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.113900 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_931f25deb4904d2fa74ab9437e733369 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:05.116904 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71cf3b95139245a4ba11b6bcaa3ab3f4 consensus state has no leader
I20250625 13:52:05.117285 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e23d05f0e5074305a2cdbd567759c6d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.117815 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e23d05f0e5074305a2cdbd567759c6d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.118883 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 11711546f0e14f9a9eabf003f49a69b7 (table default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0]) successfully deleted
W20250625 13:52:05.119127 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 already in progress: creating tablet
I20250625 13:52:05.120296 740 tablet_service.cc:1468] Processing CreateTablet for tablet 7aa06277c743464dad2ae9a396ae7dce (DEFAULT_TABLE table=default.Table_a55db7a900494ec8bb877fd245f39f60 [id=a36083ca5811443bbc0a73e3d7778c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.120455 743 tablet_service.cc:1468] Processing CreateTablet for tablet bc0f3e2955344929b261f6affa378084 (DEFAULT_TABLE table=default.Table_65b6f087b85f4a06a9131796bd8a943e [id=1f0fa3f954274c0e91ff95a6fc6dfa43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.120729 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa06277c743464dad2ae9a396ae7dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.120924 756 tablet_service.cc:1468] Processing CreateTablet for tablet d7c4a6a4fd574012818f3aa0a8b4d71f (DEFAULT_TABLE table=default.Table_65b9f8c4792145df86eba8b5f47bf1a5 [id=c9f866b3247e44279494c4e5c9502ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.121232 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c4a6a4fd574012818f3aa0a8b4d71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.123582 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9f24ae7c08bd42959ab48f7007ec98d3 (DEFAULT_TABLE table=default.Table_61c4552a00dd4dfb882f0e86f3b8776a [id=5d940360188f49d9b9e4f7b737a163c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.123762 744 tablet_service.cc:1468] Processing CreateTablet for tablet fbb6967bc38b42f7931f6a6009603d8f (DEFAULT_TABLE table=default.Table_1b3d9a83b5544828a58cdd409a9417e7 [id=e4c53996584541ccabf8915168753148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.123924 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f24ae7c08bd42959ab48f7007ec98d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.124204 751 tablet_service.cc:1468] Processing CreateTablet for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 (DEFAULT_TABLE table=default.Table_8b535d0341e94ac2b1ea25b603b076ea [id=c089310a3afc47c0b5b1086bee6d70b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.124509 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.125005 1166 tablet_bootstrap.cc:492] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.125170 1166 ts_tablet_manager.cc:1397] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.004s
I20250625 13:52:05.126006 1166 raft_consensus.cc:357] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.126158 1166 raft_consensus.cc:383] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.126211 1166 raft_consensus.cc:738] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.126477 1166 consensus_queue.cc:260] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.126638 1168 ts_tablet_manager.cc:1918] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.127525 752 tablet_service.cc:1468] Processing CreateTablet for tablet 6340de862fd04c12bb7f85557cfe53d2 (DEFAULT_TABLE table=default.Table_baa360c7bf404bb79b9e32c5d95dc8b5 [id=a1f1874f2c0d454694a40c5d5fcf4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.127736 739 tablet_service.cc:1468] Processing CreateTablet for tablet 6e2d109e6fce407dad72a2469479b5ed (DEFAULT_TABLE table=default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98 [id=e06a5b87bfb44b23beaeee29abf42788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.127828 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6340de862fd04c12bb7f85557cfe53d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.128006 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2d109e6fce407dad72a2469479b5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.128113 1166 ts_tablet_manager.cc:1428] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:05.126734 1168 log.cc:1199] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/cd4f0857f52449e1b26fde09c6c7c9d4
I20250625 13:52:05.128774 1168 ts_tablet_manager.cc:1939] T cd4f0857f52449e1b26fde09c6c7c9d4 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.129065 1166 tablet_bootstrap.cc:492] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.123582 741 tablet_service.cc:1468] Processing CreateTablet for tablet f9b82143a609453f80fcd1fa1102b30a (DEFAULT_TABLE table=default.Table_81d972e68ae9440f98ef38345914dc4d [id=6c372a1962134256a00c0f6b2a604807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.129554 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b82143a609453f80fcd1fa1102b30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.131423 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb6967bc38b42f7931f6a6009603d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.127525 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3fc555eb21fd469185828c3670006daf (DEFAULT_TABLE table=default.Table_fe7c9a0a684b4277a14d8ed100f3fa74 [id=33bfa70f7d944ff1b7e808288266b1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.133214 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc555eb21fd469185828c3670006daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.134195 1166 tablet_bootstrap.cc:654] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.135581 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0f3e2955344929b261f6affa378084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.135545 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 351) took 7384 ms. Trace:
I20250625 13:52:05.135636 1168 rpcz_store.cc:276] 0625 13:51:57.751387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.040073 (+288686us) service_pool.cc:225] Handling call
0625 13:52:05.101383 (+7061310us) ts_tablet_manager.cc:1145] Deleting tablet cd4f0857f52449e1b26fde09c6c7c9d4
0625 13:52:05.101393 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.108677 (+ 7284us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.126524 (+ 17847us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.135521 (+ 8997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7059924}
I20250625 13:52:05.137320 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 354) took 7313 ms. Trace:
I20250625 13:52:05.137429 1168 rpcz_store.cc:276] 0625 13:51:57.823483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.050248 (+226765us) service_pool.cc:225] Handling call
0625 13:52:05.136967 (+7086719us) ts_tablet_manager.cc:1145] Deleting tablet 37e4bd4aaab143ffa65a668a952ae345
0625 13:52:05.136977 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.137301 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7086372}
I20250625 13:52:05.138253 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 355) took 7309 ms. Trace:
I20250625 13:52:05.138322 1168 rpcz_store.cc:276] 0625 13:51:57.828319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.050669 (+222350us) service_pool.cc:225] Handling call
0625 13:52:05.138151 (+7087482us) ts_tablet_manager.cc:1145] Deleting tablet 68b8c8aa8036454298bf1e81bab91c01
0625 13:52:05.138160 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.138239 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7086861}
I20250625 13:52:05.139163 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 353) took 7315 ms. Trace:
I20250625 13:52:05.139243 1168 rpcz_store.cc:276] 0625 13:51:57.823384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.044652 (+221268us) service_pool.cc:225] Handling call
0625 13:52:05.139047 (+7094395us) ts_tablet_manager.cc:1145] Deleting tablet 32a57268a18f4f00a39f8360f9c90737
0625 13:52:05.139059 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.139144 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6948179}
I20250625 13:52:05.140259 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 359) took 6981 ms. Trace:
I20250625 13:52:05.140723 1168 rpcz_store.cc:276] 0625 13:51:58.159289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.198907 (+ 39618us) service_pool.cc:225] Handling call
0625 13:52:05.140142 (+6941235us) ts_tablet_manager.cc:1145] Deleting tablet db6e9db2375144b38ec43ffca9ca4c1e
0625 13:52:05.140153 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.140241 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6940923}
I20250625 13:52:05.141534 1168 tablet_replica.cc:331] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.142004 1168 raft_consensus.cc:2241] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.142534 1168 raft_consensus.cc:2270] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.143301 1168 ts_tablet_manager.cc:1905] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.148656 1710 pending_rounds.cc:77] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520611522560000 op_type: NO_OP noop_request { }
I20250625 13:52:05.149025 1710 raft_consensus.cc:2887] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:05.149318 1710 raft_consensus.cc:2270] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.150410 1710 ts_tablet_manager.cc:1905] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:05.152410 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet b7d888e628a94f939ecf2571e60b1948 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d888e628a94f939ecf2571e60b1948 already in progress: creating tablet
I20250625 13:52:05.153401 1166 tablet_bootstrap.cc:492] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.153514 1166 ts_tablet_manager.cc:1397] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250625 13:52:05.154279 1166 raft_consensus.cc:357] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.154522 1166 raft_consensus.cc:383] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.154640 1166 raft_consensus.cc:738] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.154945 1166 consensus_queue.cc:260] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.155604 1166 ts_tablet_manager.cc:1428] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:05.156672 1166 tablet_bootstrap.cc:492] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.156808 589 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1] alter complete (version 1)
I20250625 13:52:05.160310 1166 tablet_bootstrap.cc:654] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:05.162211 849 consensus_peers.cc:487] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0c5bea3b6214fa8af2792c5e9c411d9. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:05.163771 1776 raft_consensus.cc:491] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.163913 1776 raft_consensus.cc:513] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.164451 1776 leader_election.cc:290] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.164964 1168 ts_tablet_manager.cc:1918] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.165422 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 653267b468ef4958b98af0ed6a7bf4d3 (table default.Table_fd444f4e61ca4ecb9dc9ee9660e8e36b [id=6bfad46432b7473f88d77bfcfa2a4e9a]). Expected version 1 got 0
I20250625 13:52:05.166684 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "653267b468ef4958b98af0ed6a7bf4d3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.167438 711 leader_election.cc:343] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.169525 1710 ts_tablet_manager.cc:1918] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.166679 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "653267b468ef4958b98af0ed6a7bf4d3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:05.170116 1168 log.cc:1199] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/36da1da539e24f55978d516873cb0b27
W20250625 13:52:05.170188 711 leader_election.cc:343] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.170307 711 leader_election.cc:304] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:05.170704 1776 raft_consensus.cc:2747] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.171272 1168 ts_tablet_manager.cc:1939] T 36da1da539e24f55978d516873cb0b27 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.171819 1710 log.cc:1199] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/04479dfb34584f0891c604ef72bf93b5
I20250625 13:52:05.172360 1710 ts_tablet_manager.cc:1939] T 04479dfb34584f0891c604ef72bf93b5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.172469 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 360) took 7004 ms. Trace:
I20250625 13:52:05.173650 1168 rpcz_store.cc:276] 0625 13:51:58.167714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.199237 (+ 31523us) service_pool.cc:225] Handling call
0625 13:52:05.141484 (+6942247us) ts_tablet_manager.cc:1145] Deleting tablet 36da1da539e24f55978d516873cb0b27
0625 13:52:05.141495 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.154035 (+ 12540us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.164852 (+ 10817us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.172444 (+ 7592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6942075}
I20250625 13:52:05.174186 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 620) took 1737 ms. Trace:
I20250625 13:52:05.174474 1710 rpcz_store.cc:276] 0625 13:52:03.436538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.482223 (+ 45685us) service_pool.cc:225] Handling call
0625 13:52:05.106637 (+1624414us) ts_tablet_manager.cc:1145] Deleting tablet 04479dfb34584f0891c604ef72bf93b5
0625 13:52:05.106645 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.159458 (+ 52813us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.169423 (+ 9965us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.174164 (+ 4741us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1623908}
I20250625 13:52:05.175244 1710 tablet_replica.cc:331] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.175765 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 370) took 6811 ms. Trace:
I20250625 13:52:05.176172 1168 rpcz_store.cc:276] 0625 13:51:58.364210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.364282 (+ 72us) service_pool.cc:225] Handling call
0625 13:52:05.175647 (+6811365us) ts_tablet_manager.cc:1145] Deleting tablet 535812d46d6444fda85d9767c6ca7e62
0625 13:52:05.175658 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.175750 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6808364}
I20250625 13:52:05.176234 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 with delete_type TABLET_DATA_DELETED (replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.176864 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 371) took 6812 ms. Trace:
I20250625 13:52:05.177554 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cd5303e674246ada7d36574b535b333 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.177747 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a22ade8cefac4140b08882c311ea5181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.177891 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 29be8c0350584de382985e66556bdcca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.177937 746 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.178064 1165 tablet_replica.cc:331] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.178192 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd51cad7d6342d7a1d45f59653fa32f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.178236 1165 raft_consensus.cc:2241] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.178368 1165 raft_consensus.cc:2270] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.178584 757 tablet_service.cc:1515] Processing DeleteTablet for tablet c0a3ea18bab944749d45da9b3139f8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.178596 1710 raft_consensus.cc:2241] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.179051 1710 pending_rounds.cc:70] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:05.179090 1165 ts_tablet_manager.cc:1905] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.179023 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cea4a53bfa854372839e62197285c18c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.179355 1710 pending_rounds.cc:77] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520611504369664 op_type: NO_OP noop_request { }
I20250625 13:52:05.179445 1710 raft_consensus.cc:2887] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:05.179519 1710 raft_consensus.cc:2270] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:05.179742 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cea4a53bfa854372839e62197285c18c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.180675 1710 ts_tablet_manager.cc:1905] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.182682 1168 rpcz_store.cc:276] 0625 13:51:58.364300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.364513 (+ 213us) service_pool.cc:225] Handling call
0625 13:52:05.176768 (+6812255us) ts_tablet_manager.cc:1145] Deleting tablet 1ca3f3760e6e41a29d004e6afe63ec98
0625 13:52:05.176776 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.176851 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6809376}
I20250625 13:52:05.183292 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 369) took 6819 ms. Trace:
I20250625 13:52:05.183382 1168 rpcz_store.cc:276] 0625 13:51:58.364111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.364637 (+ 526us) service_pool.cc:225] Handling call
0625 13:52:05.183192 (+6818555us) ts_tablet_manager.cc:1145] Deleting tablet d43ba17ebfec4244894d1b6e23200c33
0625 13:52:05.183202 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.183279 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6815676}
I20250625 13:52:05.184330 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 372) took 6819 ms. Trace:
I20250625 13:52:05.184733 1168 rpcz_store.cc:276] 0625 13:51:58.364437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.365862 (+ 1425us) service_pool.cc:225] Handling call
0625 13:52:05.184239 (+6818377us) ts_tablet_manager.cc:1145] Deleting tablet 7506cce0ec4c4ec68f25591cb98b1252
0625 13:52:05.184248 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.184317 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6794503}
I20250625 13:52:05.185374 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 374) took 6586 ms. Trace:
I20250625 13:52:05.185736 1165 ts_tablet_manager.cc:1918] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:05.185757 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 11711546f0e14f9a9eabf003f49a69b7 because the tablet was not found. No further retry: Not found: Tablet not found: 11711546f0e14f9a9eabf003f49a69b7
I20250625 13:52:05.185839 1165 log.cc:1199] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/2cd5303e674246ada7d36574b535b333
I20250625 13:52:05.185705 1168 rpcz_store.cc:276] 0625 13:51:58.599004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.599064 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:05.185287 (+6586223us) ts_tablet_manager.cc:1145] Deleting tablet 9be75d03121f4ea2906236ac3a5b0764
0625 13:52:05.185296 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.185360 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6586009}
I20250625 13:52:05.186272 1165 ts_tablet_manager.cc:1939] T 2cd5303e674246ada7d36574b535b333 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.186475 1168 tablet_replica.cc:331] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.186920 1168 raft_consensus.cc:2241] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.187268 1168 raft_consensus.cc:2270] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.187332 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 2cd5303e674246ada7d36574b535b333 (table default.Table_09cd14059fee4f4aaadc9c4e4bd5a249 [id=a79073f0368145dcab2528663f56e9aa]) successfully deleted
I20250625 13:52:05.187654 1165 tablet_replica.cc:331] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.187882 1165 raft_consensus.cc:2241] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.188026 1165 raft_consensus.cc:2270] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.192128 1165 ts_tablet_manager.cc:1905] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.192196 1166 tablet_bootstrap.cc:492] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.192370 1166 ts_tablet_manager.cc:1397] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.004s
I20250625 13:52:05.193120 1166 raft_consensus.cc:357] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.193328 1166 raft_consensus.cc:383] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.193413 1166 raft_consensus.cc:738] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.193717 1166 consensus_queue.cc:260] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.195116 1166 ts_tablet_manager.cc:1428] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:05.195781 1168 ts_tablet_manager.cc:1905] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.196331 1166 tablet_bootstrap.cc:492] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.198058 1166 tablet_bootstrap.cc:654] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.201227 1710 ts_tablet_manager.cc:1918] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.203481 1710 log.cc:1199] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/7d361e73fe3d4ea2bb6735f814345ebb
I20250625 13:52:05.204466 1710 ts_tablet_manager.cc:1939] T 7d361e73fe3d4ea2bb6735f814345ebb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.205782 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 621) took 1576 ms. Trace:
I20250625 13:52:05.206787 1710 rpcz_store.cc:276] 0625 13:52:03.629815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.631850 (+ 2035us) service_pool.cc:225] Handling call
0625 13:52:05.175195 (+1543345us) ts_tablet_manager.cc:1145] Deleting tablet 7d361e73fe3d4ea2bb6735f814345ebb
0625 13:52:05.175205 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.192126 (+ 16921us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.201114 (+ 8988us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.205759 (+ 4645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511912}
I20250625 13:52:05.207226 1168 ts_tablet_manager.cc:1918] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.207772 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 631) took 1393 ms. Trace:
I20250625 13:52:05.211361 1710 rpcz_store.cc:276] 0625 13:52:03.814051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.814122 (+ 71us) service_pool.cc:225] Handling call
0625 13:52:05.207649 (+1393527us) ts_tablet_manager.cc:1145] Deleting tablet 3c6dd5b87ff34151836a3c29740e7c11
0625 13:52:05.207660 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.207758 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371366}
I20250625 13:52:05.212186 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 632) took 1380 ms. Trace:
I20250625 13:52:05.212591 1710 rpcz_store.cc:276] 0625 13:52:03.831238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.831750 (+ 512us) service_pool.cc:225] Handling call
0625 13:52:05.212070 (+1380320us) ts_tablet_manager.cc:1145] Deleting tablet 7db450dc63bd4b6caf074c67490cd175
0625 13:52:05.212080 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.212171 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358248}
I20250625 13:52:05.213272 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 633) took 1378 ms. Trace:
I20250625 13:52:05.213626 1710 rpcz_store.cc:276] 0625 13:52:03.835116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.835963 (+ 847us) service_pool.cc:225] Handling call
0625 13:52:05.213181 (+1377218us) ts_tablet_manager.cc:1145] Deleting tablet 455884b62ee44b6da53543093f721027
0625 13:52:05.213192 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.213260 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350011}
I20250625 13:52:05.214071 1165 ts_tablet_manager.cc:1918] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.214361 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 635) took 1359 ms. Trace:
I20250625 13:52:05.214784 1165 log.cc:1199] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a22ade8cefac4140b08882c311ea5181
I20250625 13:52:05.214733 1710 rpcz_store.cc:276] 0625 13:52:03.855270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.856100 (+ 830us) service_pool.cc:225] Handling call
0625 13:52:05.214261 (+1358161us) ts_tablet_manager.cc:1145] Deleting tablet be3313ec5dfb41b68662d3ff4c8c94f9
0625 13:52:05.214268 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.214345 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1284696}
I20250625 13:52:05.215111 1165 ts_tablet_manager.cc:1939] T a22ade8cefac4140b08882c311ea5181 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.216282 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a22ade8cefac4140b08882c311ea5181 (table default.Table_9dfb8c48d5144c808567a16a30cc199e [id=f55ecdcac2234415a085be00cacad571]) successfully deleted
I20250625 13:52:05.217309 1168 log.cc:1199] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/055f88e118454e32baa578cae9aceb0c
I20250625 13:52:05.217700 1168 ts_tablet_manager.cc:1939] T 055f88e118454e32baa578cae9aceb0c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.218055 1165 tablet_replica.cc:331] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.218461 1165 raft_consensus.cc:2241] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.219890 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 382) took 6462 ms. Trace:
I20250625 13:52:05.219991 1168 rpcz_store.cc:276] 0625 13:51:58.757718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.757784 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:05.186428 (+6428644us) ts_tablet_manager.cc:1145] Deleting tablet 055f88e118454e32baa578cae9aceb0c
0625 13:52:05.186439 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.204302 (+ 17863us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.207137 (+ 2835us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.219867 (+ 12730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6428349}
I20250625 13:52:05.224995 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b1acc41a93f6433588727811903921fe
I20250625 13:52:05.225379 596 catalog_manager.cc:6652] ReplaceTablet: tablet b1acc41a93f6433588727811903921fe deleted and replaced by tablet d5cb3b35a5c2483fa82226133a7c161d
I20250625 13:52:05.225772 1165 raft_consensus.cc:2270] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.226706 1165 ts_tablet_manager.cc:1905] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.227697 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet da2f8d0d9a714a28ba6fb9642d461de2
I20250625 13:52:05.229238 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 00d4acd387c84d55834f632f41045862
I20250625 13:52:05.232486 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 634) took 1383 ms. Trace:
I20250625 13:52:05.232775 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 58a532e889244b088562081e444234a3
I20250625 13:52:05.233511 592 catalog_manager.cc:6652] ReplaceTablet: tablet 58a532e889244b088562081e444234a3 deleted and replaced by tablet 1c4a6d7d0c9647a4a17dca28d6a063a5
W20250625 13:52:05.221783 849 consensus_peers.cc:487] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c911d9ecbded417daf70e85c7cbcd52e. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:05.237536 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 384) took 6475 ms. Trace:
I20250625 13:52:05.237632 1168 rpcz_store.cc:276] 0625 13:51:58.761631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.761844 (+ 213us) service_pool.cc:225] Handling call
0625 13:52:05.237420 (+6475576us) ts_tablet_manager.cc:1145] Deleting tablet 3249c5c2aa564027a2c594ca4c31e1c8
0625 13:52:05.237430 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.237519 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6475367}
I20250625 13:52:05.238003 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 385) took 6476 ms. Trace:
I20250625 13:52:05.238075 1168 rpcz_store.cc:276] 0625 13:51:58.761789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.762090 (+ 301us) service_pool.cc:225] Handling call
0625 13:52:05.237895 (+6475805us) ts_tablet_manager.cc:1145] Deleting tablet 0b129117db064a15b27e82439637cf22
0625 13:52:05.237904 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.237986 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6475612}
I20250625 13:52:05.238366 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 383) took 6476 ms. Trace:
I20250625 13:52:05.238440 1168 rpcz_store.cc:276] 0625 13:51:58.761510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.762324 (+ 814us) service_pool.cc:225] Handling call
0625 13:52:05.238274 (+6475950us) ts_tablet_manager.cc:1145] Deleting tablet d9462da7574a46839763b1c388102e60
0625 13:52:05.238282 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.238351 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6475748}
I20250625 13:52:05.238746 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 386) took 6465 ms. Trace:
I20250625 13:52:05.238813 1168 rpcz_store.cc:276] 0625 13:51:58.772922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.772977 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:05.238658 (+6465681us) ts_tablet_manager.cc:1145] Deleting tablet 38a23b08a03b48eaa051f670eb7dac65
0625 13:52:05.238667 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.238731 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6465468}
I20250625 13:52:05.239056 1168 tablet_replica.cc:331] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.239238 1168 raft_consensus.cc:2241] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.232590 1710 rpcz_store.cc:276] 0625 13:52:03.849504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.849629 (+ 125us) service_pool.cc:225] Handling call
0625 13:52:05.215731 (+1366102us) ts_tablet_manager.cc:1145] Deleting tablet 4d25e4114a184bc99705042593ab4ab9
0625 13:52:05.215738 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.232458 (+ 16720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1284913}
I20250625 13:52:05.240288 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 58a532e889244b088562081e444234a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c4a6d7d0c9647a4a17dca28d6a063a5) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.240638 1842 raft_consensus.cc:491] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.240705 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b1acc41a93f6433588727811903921fe with delete_type TABLET_DATA_DELETED (replaced by tablet d5cb3b35a5c2483fa82226133a7c161d) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.240804 1842 raft_consensus.cc:513] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.240872 741 tablet_service.cc:1515] Processing DeleteTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.241240 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d4acd387c84d55834f632f41045862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.241561 1842 leader_election.cc:290] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:05.242228 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e170ed64e7174c39a3e21ceac0e51021" } new_table_name: "default.Table_da4c3565bbf845ca9a26fe3d1f39f99f" modify_external_catalogs: true
I20250625 13:52:05.242985 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae4a105e80c1460cbb21e1bdb9b356b1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:05.243476 1710 tablet_replica.cc:331] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.243685 1710 raft_consensus.cc:2241] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.243680 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dcb69af0d4fc42a5982d04bc02404a49" } modify_external_catalogs: true
I20250625 13:52:05.243866 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_dcb69af0d4fc42a5982d04bc02404a49" } modify_external_catalogs: true
I20250625 13:52:05.243930 1710 pending_rounds.cc:70] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
W20250625 13:52:05.243851 851 leader_election.cc:343] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ae4a105e80c1460cbb21e1bdb9b356b1
I20250625 13:52:05.243968 1710 pending_rounds.cc:77] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520612784873472 op_type: NO_OP noop_request { }
I20250625 13:52:05.244071 1710 raft_consensus.cc:2887] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:05.244165 1710 raft_consensus.cc:2270] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:05.244150 849 leader_election.cc:343] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.244244 849 leader_election.cc:304] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.245016 1842 raft_consensus.cc:2747] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.246031 596 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
I20250625 13:52:05.246604 1710 ts_tablet_manager.cc:1905] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.247145 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5821b12a3aff40dcb22e592583f8c1f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.250085 756 tablet_service.cc:1515] Processing DeleteTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.250384 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5821b12a3aff40dcb22e592583f8c1f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.251180 1045 raft_consensus.cc:3058] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.251703 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_71166441017840fa9a67752209689ee2" } modify_external_catalogs: true
I20250625 13:52:05.251889 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_71166441017840fa9a67752209689ee2" } modify_external_catalogs: true
I20250625 13:52:05.252091 1166 tablet_bootstrap.cc:492] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.252206 1166 ts_tablet_manager.cc:1397] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.015s
I20250625 13:52:05.252187 1168 raft_consensus.cc:2270] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.252751 1168 ts_tablet_manager.cc:1905] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.252995 1166 raft_consensus.cc:357] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.253031 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0949521f201f43578a2039b556e84501" } modify_external_catalogs: true
I20250625 13:52:05.253131 1166 raft_consensus.cc:383] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.253172 1166 raft_consensus.cc:738] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.253166 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0949521f201f43578a2039b556e84501" } modify_external_catalogs: true
I20250625 13:52:05.253360 1166 consensus_queue.cc:260] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.253945 1166 ts_tablet_manager.cc:1428] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:05.254205 1166 tablet_bootstrap.cc:492] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.254784 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c4db530b54f04fedb18bf993dc60d8da
I20250625 13:52:05.255576 593 catalog_manager.cc:6652] ReplaceTablet: tablet c4db530b54f04fedb18bf993dc60d8da deleted and replaced by tablet c366da4fb138453386d97fba0f5c91be
I20250625 13:52:05.256438 741 tablet_service.cc:1515] Processing DeleteTablet for tablet c4db530b54f04fedb18bf993dc60d8da with delete_type TABLET_DATA_DELETED (replaced by tablet c366da4fb138453386d97fba0f5c91be) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.256832 1166 tablet_bootstrap.cc:654] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.259518 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_abea1e01b70d4edeb3cfbbd851205b11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.260075 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abea1e01b70d4edeb3cfbbd851205b11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.261452 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_29c1207ed89e4a33bceb77bc37580565"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.262028 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29c1207ed89e4a33bceb77bc37580565 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.260546 1836 raft_consensus.cc:491] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.262369 1836 raft_consensus.cc:513] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.262696 1710 ts_tablet_manager.cc:1918] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.264777 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d8906d18878c49f59fe57ee84374beb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.265367 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8906d18878c49f59fe57ee84374beb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.265477 1776 raft_consensus.cc:491] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.265653 1776 raft_consensus.cc:513] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.266265 1776 leader_election.cc:290] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.266880 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f065b4e3218b4e6e8a2ac3a70eede13b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:05.267419 711 leader_election.cc:343] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.262995 1836 leader_election.cc:290] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:05.264036 985 leader_election.cc:343] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 055f88e118454e32baa578cae9aceb0c
W20250625 13:52:05.268653 983 leader_election.cc:343] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 055f88e118454e32baa578cae9aceb0c
I20250625 13:52:05.268710 1710 log.cc:1199] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/cd4f0857f52449e1b26fde09c6c7c9d4
I20250625 13:52:05.268852 983 leader_election.cc:304] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:05.269078 1710 ts_tablet_manager.cc:1939] T cd4f0857f52449e1b26fde09c6c7c9d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.269243 1836 raft_consensus.cc:2747] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.267048 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f065b4e3218b4e6e8a2ac3a70eede13b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.269829 711 leader_election.cc:343] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.269949 711 leader_election.cc:304] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:05.270367 1776 raft_consensus.cc:2747] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.270802 1168 ts_tablet_manager.cc:1918] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.272300 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_40856c021b18443fb5feae09a82e09ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.272416 756 tablet_service.cc:1515] Processing DeleteTablet for tablet abd6f9121a9d4b39adde1b880a875e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.272946 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40856c021b18443fb5feae09a82e09ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.274797 1710 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 646) took 1224 ms. Trace:
I20250625 13:52:05.275533 1710 rpcz_store.cc:276] 0625 13:52:04.050389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.074102 (+ 23713us) service_pool.cc:225] Handling call
0625 13:52:05.243423 (+1169321us) ts_tablet_manager.cc:1145] Deleting tablet cd4f0857f52449e1b26fde09c6c7c9d4
0625 13:52:05.243436 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.249712 (+ 6276us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.262598 (+ 12886us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.274773 (+ 12175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1168302}
I20250625 13:52:05.279883 1710 tablet_replica.cc:331] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.283499 1710 raft_consensus.cc:2241] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.284025 1710 raft_consensus.cc:2270] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.284116 1165 ts_tablet_manager.cc:1918] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.284310 1165 log.cc:1199] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/29be8c0350584de382985e66556bdcca
I20250625 13:52:05.281745 1895 tablet.cc:1722] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:05.284853 1710 ts_tablet_manager.cc:1905] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.288856 1165 ts_tablet_manager.cc:1939] T 29be8c0350584de382985e66556bdcca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.289884 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 29be8c0350584de382985e66556bdcca (table default.Table_13c3269d65ff41bc87efff42b2dce296 [id=1c58b83050f1494a91b0845e8dfe04bf]) successfully deleted
I20250625 13:52:05.290598 1165 tablet_replica.cc:331] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.291038 1165 raft_consensus.cc:2241] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.291482 1165 raft_consensus.cc:2270] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.292735 1165 ts_tablet_manager.cc:1905] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.295403 1168 log.cc:1199] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/600d711975a74f66b9144758ad0dcfb0
W20250625 13:52:05.296456 581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 253916us
I20250625 13:52:05.296833 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 27092991fcad475fabc77d1f8bab0582 (table default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba [id=540530d4619a4093b20d1bcba9ba99ac]) successfully deleted
W20250625 13:52:05.297183 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a0e81d4c904248eba858462bcf31c5d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0e81d4c904248eba858462bcf31c5d9 already in progress: creating tablet
W20250625 13:52:05.297585 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1effed0fa10d409abcb31dbba54861d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1effed0fa10d409abcb31dbba54861d6 already in progress: creating tablet
I20250625 13:52:05.297857 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 02f2a2b595bc43309a29b3f1bc5d3766 (table default.Table_630c47cb56984efbac74836e412efaa6 [id=8c054766f5424144a4edb39b192451cb]) successfully deleted
I20250625 13:52:05.298065 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet cd4f0857f52449e1b26fde09c6c7c9d4 (table default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]) successfully deleted
W20250625 13:52:05.298342 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 37e4bd4aaab143ffa65a668a952ae345 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37e4bd4aaab143ffa65a668a952ae345 already in progress: creating tablet
W20250625 13:52:05.299181 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 68b8c8aa8036454298bf1e81bab91c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68b8c8aa8036454298bf1e81bab91c01 already in progress: creating tablet
I20250625 13:52:05.299846 1165 ts_tablet_manager.cc:1918] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.301719 1168 ts_tablet_manager.cc:1939] T 600d711975a74f66b9144758ad0dcfb0 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.303093 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 389) took 6527 ms. Trace:
I20250625 13:52:05.304159 1168 rpcz_store.cc:276] 0625 13:51:58.775673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.775723 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:05.239012 (+6463289us) ts_tablet_manager.cc:1145] Deleting tablet 600d711975a74f66b9144758ad0dcfb0
0625 13:52:05.239020 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.267863 (+ 28843us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.270717 (+ 2854us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.303069 (+ 32352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6463091}
I20250625 13:52:05.305063 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 388) took 6530 ms. Trace:
I20250625 13:52:05.305999 1168 rpcz_store.cc:276] 0625 13:51:58.774890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.778456 (+ 3566us) service_pool.cc:225] Handling call
0625 13:52:05.304938 (+6526482us) ts_tablet_manager.cc:1145] Deleting tablet a30a35e3639a43c5b078169060824723
0625 13:52:05.304949 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.305046 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6525407}
I20250625 13:52:05.306677 1168 tablet_replica.cc:331] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.308162 1165 log.cc:1199] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ecb9a98e402b4da8a9c2bd8317cef532
I20250625 13:52:05.308738 1165 ts_tablet_manager.cc:1939] T ecb9a98e402b4da8a9c2bd8317cef532 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.310254 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ecb9a98e402b4da8a9c2bd8317cef532 (table default.Table_cc7e074a01254cbb9272fd4ea7dba124 [id=d85d6e75f7e9420096456eac1ab144cc]) successfully deleted
I20250625 13:52:05.311255 1165 tablet_replica.cc:331] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.311578 1168 raft_consensus.cc:2241] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
W20250625 13:52:05.311693 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 32a57268a18f4f00a39f8360f9c90737 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a57268a18f4f00a39f8360f9c90737 already in progress: creating tablet
I20250625 13:52:05.311769 1165 raft_consensus.cc:2241] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:05.312031 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet db6e9db2375144b38ec43ffca9ca4c1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db6e9db2375144b38ec43ffca9ca4c1e already in progress: creating tablet
I20250625 13:52:05.312194 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 36da1da539e24f55978d516873cb0b27 (table default.Table_b079a0f225cf4ac98c2efc03bb74b889 [id=83a712e3adf74a95a5c2ea3fc5c0e14e]) successfully deleted
I20250625 13:52:05.312254 1168 raft_consensus.cc:2270] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.312260 1165 raft_consensus.cc:2270] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:05.312423 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 535812d46d6444fda85d9767c6ca7e62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 535812d46d6444fda85d9767c6ca7e62 already in progress: creating tablet
W20250625 13:52:05.312577 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1ca3f3760e6e41a29d004e6afe63ec98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ca3f3760e6e41a29d004e6afe63ec98 already in progress: creating tablet
W20250625 13:52:05.312757 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d43ba17ebfec4244894d1b6e23200c33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43ba17ebfec4244894d1b6e23200c33 already in progress: creating tablet
W20250625 13:52:05.312919 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 7506cce0ec4c4ec68f25591cb98b1252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7506cce0ec4c4ec68f25591cb98b1252 already in progress: creating tablet
W20250625 13:52:05.313104 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9be75d03121f4ea2906236ac3a5b0764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9be75d03121f4ea2906236ac3a5b0764 already in progress: creating tablet
I20250625 13:52:05.313294 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 055f88e118454e32baa578cae9aceb0c (table default.Table_df6a42eba40144cf94d09e54694a85b9 [id=e50f8ace632e4ea284d17b85f474cef5]) successfully deleted
W20250625 13:52:05.313458 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3249c5c2aa564027a2c594ca4c31e1c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3249c5c2aa564027a2c594ca4c31e1c8 already in progress: creating tablet
W20250625 13:52:05.313602 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 0b129117db064a15b27e82439637cf22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b129117db064a15b27e82439637cf22 already in progress: creating tablet
W20250625 13:52:05.313786 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d9462da7574a46839763b1c388102e60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9462da7574a46839763b1c388102e60 already in progress: creating tablet
W20250625 13:52:05.313956 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 38a23b08a03b48eaa051f670eb7dac65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38a23b08a03b48eaa051f670eb7dac65 already in progress: creating tablet
I20250625 13:52:05.314101 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 600d711975a74f66b9144758ad0dcfb0 (table default.Table_186697102c744ac39ec9191fc0f992ea [id=548110492fba410e9612ddd9792eed22]) successfully deleted
W20250625 13:52:05.314245 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a30a35e3639a43c5b078169060824723 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a30a35e3639a43c5b078169060824723 already in progress: creating tablet
I20250625 13:52:05.314716 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 254afffd283449fdaf5e190ba3028f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.316556 1168 ts_tablet_manager.cc:1905] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.318215 1165 ts_tablet_manager.cc:1905] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.322502 1171 tablet_bootstrap.cc:492] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.322727 1171 ts_tablet_manager.cc:1397] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.225s user 0.035s sys 0.020s
I20250625 13:52:05.322804 1776 raft_consensus.cc:491] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.322934 1776 raft_consensus.cc:513] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:05.323285 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81a359c798024b2a99aa3b307cc1967f consensus state has no leader
I20250625 13:52:05.323657 1776 leader_election.cc:290] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.324447 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd6f9121a9d4b39adde1b880a875e23" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:05.324879 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd6f9121a9d4b39adde1b880a875e23" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.325004 711 leader_election.cc:343] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.325273 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5426c9800c16492fa6da1e38ccdf5139" } new_table_name: "default.Table_c9744a7c82cf465696fd5127da2131ba" modify_external_catalogs: true
W20250625 13:52:05.325451 711 leader_election.cc:343] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.325558 711 leader_election.cc:304] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:05.325816 1710 ts_tablet_manager.cc:1918] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.325909 1710 log.cc:1199] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/bb1c7a85da844d34be805843bc60d350
I20250625 13:52:05.326107 1776 raft_consensus.cc:2747] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.326287 1710 ts_tablet_manager.cc:1939] T bb1c7a85da844d34be805843bc60d350 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.326741 1171 raft_consensus.cc:357] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.327042 1171 raft_consensus.cc:383] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.327160 1171 raft_consensus.cc:738] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.327505 1171 consensus_queue.cc:260] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.330554 1842 consensus_queue.cc:1035] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:05.333444 1710 tablet_replica.cc:331] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.333773 1710 raft_consensus.cc:2241] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.333988 1710 raft_consensus.cc:2270] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.334661 1710 ts_tablet_manager.cc:1905] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.337733 588 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet e666fcf8688742cca895a8692d1b2458
I20250625 13:52:05.339973 1171 ts_tablet_manager.cc:1428] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 13:52:05.342252 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9516364563e3471dbcbf1541c7ff5bd1" } new_table_name: "default.Table_39575a8dc48640dbae773650f2417b76" modify_external_catalogs: true
I20250625 13:52:05.342715 1171 tablet_bootstrap.cc:492] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.345271 1168 ts_tablet_manager.cc:1918] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.346578 1170 tablet_bootstrap.cc:492] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:05.346751 1170 ts_tablet_manager.cc:1397] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.272s user 0.028s sys 0.020s
I20250625 13:52:05.347623 1170 raft_consensus.cc:357] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.347821 1170 raft_consensus.cc:383] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.347915 1170 raft_consensus.cc:738] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.348132 1170 consensus_queue.cc:260] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.348862 1170 ts_tablet_manager.cc:1428] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:05.349814 1170 tablet_bootstrap.cc:492] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:05.352138 1170 tablet_bootstrap.cc:654] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.353623 1168 log.cc:1199] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/8149aea0ff7646d3b94c9159f56e48ef
I20250625 13:52:05.353874 1165 ts_tablet_manager.cc:1918] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.354138 1168 ts_tablet_manager.cc:1939] T 8149aea0ff7646d3b94c9159f56e48ef P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.355846 1836 raft_consensus.cc:491] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.355983 1836 raft_consensus.cc:513] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.356673 1836 leader_election.cc:290] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.356899 1165 log.cc:1199] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/2bbd509e3dfb45d98314c6afe37677f6
I20250625 13:52:05.357398 1165 ts_tablet_manager.cc:1939] T 2bbd509e3dfb45d98314c6afe37677f6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:52:05.357617 985 leader_election.cc:343] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f033bfa4925b4d1c971789eb77efd92c
I20250625 13:52:05.358464 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f5d01b07d90441a79d2ccd6c8fa92610" } new_table_name: "default.Table_8190e518ba344ab9b9081933923dae13" modify_external_catalogs: true
I20250625 13:52:05.361689 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 254afffd283449fdaf5e190ba3028f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.361955 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f033bfa4925b4d1c971789eb77efd92c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:05.362208 912 raft_consensus.cc:2466] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:05.362373 1166 tablet_bootstrap.cc:492] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.362413 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.362434 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 387) took 6587 ms. Trace:
I20250625 13:52:05.362501 1166 ts_tablet_manager.cc:1397] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.108s user 0.009s sys 0.013s
I20250625 13:52:05.362560 1168 rpcz_store.cc:276] 0625 13:51:58.774736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.779293 (+ 4557us) service_pool.cc:225] Handling call
0625 13:52:05.306631 (+6527338us) ts_tablet_manager.cc:1145] Deleting tablet 8149aea0ff7646d3b94c9159f56e48ef
0625 13:52:05.306641 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.322117 (+ 15476us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.345129 (+ 23012us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.362400 (+ 17271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6526970}
I20250625 13:52:05.362918 1168 tablet_replica.cc:331] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.363130 1168 raft_consensus.cc:2241] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.363293 1168 raft_consensus.cc:2270] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.363245 1166 raft_consensus.cc:357] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.363494 1166 raft_consensus.cc:383] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.363610 1166 raft_consensus.cc:738] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.363869 1166 consensus_queue.cc:260] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.364040 983 leader_election.cc:304] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.364401 1166 ts_tablet_manager.cc:1428] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:05.364634 1166 tablet_bootstrap.cc:492] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.365185 1836 raft_consensus.cc:2802] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:05.365265 1836 raft_consensus.cc:491] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:05.365345 1836 raft_consensus.cc:3058] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:05.363831 849 consensus_peers.cc:487] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15e58fbec94e4ff9b7ac9830b576fe7f. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:05.367100 1166 tablet_bootstrap.cc:654] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.367599 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f065b4e3218b4e6e8a2ac3a70eede13b
I20250625 13:52:05.368249 1836 raft_consensus.cc:513] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.368871 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 2bbd509e3dfb45d98314c6afe37677f6 (table default.Table_683f20e141ca4122bee28d4e86aece20 [id=40f54158a1c84f5988762e7a44e3eab1]) successfully deleted
I20250625 13:52:05.369279 756 tablet_service.cc:1515] Processing DeleteTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.370242 985 leader_election.cc:343] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f033bfa4925b4d1c971789eb77efd92c
I20250625 13:52:05.371948 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f033bfa4925b4d1c971789eb77efd92c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:05.372160 912 raft_consensus.cc:3058] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.374531 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_734235bd5dc045678fdad3d130384a52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.374929 912 raft_consensus.cc:2466] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:05.375108 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_734235bd5dc045678fdad3d130384a52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.375630 983 leader_election.cc:304] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.375908 1821 raft_consensus.cc:2802] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:05.367379 1168 ts_tablet_manager.cc:1905] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.379276 1836 leader_election.cc:290] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.382153 891 tablet_service.cc:1515] Processing DeleteTablet for tablet b7d888e628a94f939ecf2571e60b1948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.384061 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet b7d888e628a94f939ecf2571e60b1948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.384477 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.386842 1165 tablet_replica.cc:331] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.387145 1165 raft_consensus.cc:2241] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.387419 1165 raft_consensus.cc:2270] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.388334 1165 ts_tablet_manager.cc:1905] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.388454 1019 tablet_service.cc:1468] Processing CreateTablet for tablet bc0f3e2955344929b261f6affa378084 (DEFAULT_TABLE table=default.Table_65b6f087b85f4a06a9131796bd8a943e [id=1f0fa3f954274c0e91ff95a6fc6dfa43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.388840 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0f3e2955344929b261f6affa378084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.346066 1171 tablet_bootstrap.cc:654] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.396368 1171 tablet_bootstrap.cc:492] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.396593 1171 ts_tablet_manager.cc:1397] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.003s
I20250625 13:52:05.396966 884 tablet_service.cc:1468] Processing CreateTablet for tablet bc0f3e2955344929b261f6affa378084 (DEFAULT_TABLE table=default.Table_65b6f087b85f4a06a9131796bd8a943e [id=1f0fa3f954274c0e91ff95a6fc6dfa43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.397388 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0f3e2955344929b261f6affa378084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.397480 1171 raft_consensus.cc:357] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.397681 1171 raft_consensus.cc:383] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.397763 1171 raft_consensus.cc:738] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.397995 1171 consensus_queue.cc:260] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.398506 1171 ts_tablet_manager.cc:1428] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:05.398576 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d7c4a6a4fd574012818f3aa0a8b4d71f (DEFAULT_TABLE table=default.Table_65b9f8c4792145df86eba8b5f47bf1a5 [id=c9f866b3247e44279494c4e5c9502ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.398934 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c4a6a4fd574012818f3aa0a8b4d71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.399739 1165 ts_tablet_manager.cc:1918] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.403378 891 tablet_service.cc:1468] Processing CreateTablet for tablet d7c4a6a4fd574012818f3aa0a8b4d71f (DEFAULT_TABLE table=default.Table_65b9f8c4792145df86eba8b5f47bf1a5 [id=c9f866b3247e44279494c4e5c9502ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.403648 1171 tablet_bootstrap.cc:492] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.404289 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 991bb428def545fd979e9f1a80ce4d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.404521 1165 log.cc:1199] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/c0a3ea18bab944749d45da9b3139f8bc
I20250625 13:52:05.404954 1023 tablet_service.cc:1468] Processing CreateTablet for tablet f9b82143a609453f80fcd1fa1102b30a (DEFAULT_TABLE table=default.Table_81d972e68ae9440f98ef38345914dc4d [id=6c372a1962134256a00c0f6b2a604807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.405400 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b82143a609453f80fcd1fa1102b30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.406603 1171 tablet_bootstrap.cc:654] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.406738 1165 ts_tablet_manager.cc:1939] T c0a3ea18bab944749d45da9b3139f8bc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.409963 1165 tablet_replica.cc:331] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.410195 1165 raft_consensus.cc:2241] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.410421 1165 raft_consensus.cc:2270] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.410751 1710 ts_tablet_manager.cc:1918] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.410962 1710 log.cc:1199] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/643928ca4fc94ce58431a0525f2abb99
I20250625 13:52:05.411382 1710 ts_tablet_manager.cc:1939] T 643928ca4fc94ce58431a0525f2abb99 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.411552 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c4a6a4fd574012818f3aa0a8b4d71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.412374 1710 tablet_replica.cc:331] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.416050 1821 raft_consensus.cc:695] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:05.416421 1821 consensus_queue.cc:237] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:05.416731 849 consensus_peers.cc:487] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:05.417697 1165 ts_tablet_manager.cc:1905] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.421761 1166 tablet_bootstrap.cc:492] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.421940 1166 ts_tablet_manager.cc:1397] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.004s
I20250625 13:52:05.422397 1168 ts_tablet_manager.cc:1918] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.422535 1168 log.cc:1199] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/b7641869a0bb463698244ff445d87a1f
I20250625 13:52:05.422869 1168 ts_tablet_manager.cc:1939] T b7641869a0bb463698244ff445d87a1f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.422897 1166 raft_consensus.cc:357] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.423177 1166 raft_consensus.cc:383] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.423275 1166 raft_consensus.cc:738] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.423615 1166 consensus_queue.cc:260] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.425577 1166 ts_tablet_manager.cc:1428] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:05.425890 1166 tablet_bootstrap.cc:492] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.429649 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 394) took 6572 ms. Trace:
I20250625 13:52:05.430303 1168 rpcz_store.cc:276] 0625 13:51:58.857083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.857159 (+ 76us) service_pool.cc:225] Handling call
0625 13:52:05.362840 (+6505681us) ts_tablet_manager.cc:1145] Deleting tablet b7641869a0bb463698244ff445d87a1f
0625 13:52:05.362851 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.416924 (+ 54073us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.422296 (+ 5372us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.429587 (+ 7291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6500205}
I20250625 13:52:05.430462 1710 raft_consensus.cc:2241] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.430917 1710 pending_rounds.cc:70] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:05.431042 1710 pending_rounds.cc:77] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520616108568576 op_type: NO_OP noop_request { }
I20250625 13:52:05.431286 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 396) took 6497 ms. Trace:
I20250625 13:52:05.431892 1710 raft_consensus.cc:2887] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:05.432814 1710 raft_consensus.cc:2270] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.433733 1710 ts_tablet_manager.cc:1905] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.435438 1168 rpcz_store.cc:276] 0625 13:51:58.933917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:58.933972 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:05.431126 (+6497154us) ts_tablet_manager.cc:1145] Deleting tablet ccce48011a15405db2b95c04c9b01b3c
0625 13:52:05.431136 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.431235 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6479056}
I20250625 13:52:05.436275 1168 tablet_replica.cc:331] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.437472 1168 raft_consensus.cc:2241] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.437990 1168 raft_consensus.cc:2270] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.439055 1168 ts_tablet_manager.cc:1905] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.440251 1166 tablet_bootstrap.cc:654] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.440107 1710 ts_tablet_manager.cc:1918] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.442098 1710 log.cc:1199] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/f033bfa4925b4d1c971789eb77efd92c
I20250625 13:52:05.442461 1710 ts_tablet_manager.cc:1939] T f033bfa4925b4d1c971789eb77efd92c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.444790 1023 tablet_service.cc:1468] Processing CreateTablet for tablet fbb6967bc38b42f7931f6a6009603d8f (DEFAULT_TABLE table=default.Table_1b3d9a83b5544828a58cdd409a9417e7 [id=e4c53996584541ccabf8915168753148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.445205 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb6967bc38b42f7931f6a6009603d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.440135 1171 tablet_bootstrap.cc:492] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.446367 1171 ts_tablet_manager.cc:1397] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.003s
I20250625 13:52:05.447433 1171 raft_consensus.cc:357] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.447687 1171 raft_consensus.cc:383] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.447794 1171 raft_consensus.cc:738] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.449136 1171 consensus_queue.cc:260] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.449755 1171 ts_tablet_manager.cc:1428] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:05.443522 891 tablet_service.cc:1468] Processing CreateTablet for tablet f9b82143a609453f80fcd1fa1102b30a (DEFAULT_TABLE table=default.Table_81d972e68ae9440f98ef38345914dc4d [id=6c372a1962134256a00c0f6b2a604807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.451761 1171 tablet_bootstrap.cc:492] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.452978 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b82143a609453f80fcd1fa1102b30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.455195 1171 tablet_bootstrap.cc:654] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.456133 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 (DEFAULT_TABLE table=default.Table_8b535d0341e94ac2b1ea25b603b076ea [id=c089310a3afc47c0b5b1086bee6d70b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.456118 884 tablet_service.cc:1468] Processing CreateTablet for tablet fbb6967bc38b42f7931f6a6009603d8f (DEFAULT_TABLE table=default.Table_1b3d9a83b5544828a58cdd409a9417e7 [id=e4c53996584541ccabf8915168753148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.456481 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.456501 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb6967bc38b42f7931f6a6009603d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.467032 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 7aa06277c743464dad2ae9a396ae7dce (DEFAULT_TABLE table=default.Table_a55db7a900494ec8bb877fd245f39f60 [id=a36083ca5811443bbc0a73e3d7778c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.467422 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa06277c743464dad2ae9a396ae7dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.467933 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 (DEFAULT_TABLE table=default.Table_8b535d0341e94ac2b1ea25b603b076ea [id=c089310a3afc47c0b5b1086bee6d70b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.468282 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.468187 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6340de862fd04c12bb7f85557cfe53d2 (DEFAULT_TABLE table=default.Table_baa360c7bf404bb79b9e32c5d95dc8b5 [id=a1f1874f2c0d454694a40c5d5fcf4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.468603 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6340de862fd04c12bb7f85557cfe53d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.473621 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 6340de862fd04c12bb7f85557cfe53d2 (DEFAULT_TABLE table=default.Table_baa360c7bf404bb79b9e32c5d95dc8b5 [id=a1f1874f2c0d454694a40c5d5fcf4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.473992 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6340de862fd04c12bb7f85557cfe53d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.480237 1170 tablet_bootstrap.cc:492] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:05.480372 1170 ts_tablet_manager.cc:1397] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.131s user 0.008s sys 0.006s
I20250625 13:52:05.480729 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 9f24ae7c08bd42959ab48f7007ec98d3 (DEFAULT_TABLE table=default.Table_61c4552a00dd4dfb882f0e86f3b8776a [id=5d940360188f49d9b9e4f7b737a163c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.481130 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f24ae7c08bd42959ab48f7007ec98d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.481470 1170 raft_consensus.cc:357] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.481729 1170 raft_consensus.cc:383] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.481799 1170 raft_consensus.cc:738] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.482108 1170 consensus_queue.cc:260] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.484367 1170 ts_tablet_manager.cc:1428] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:05.484772 1170 tablet_bootstrap.cc:492] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:05.487929 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7aa06277c743464dad2ae9a396ae7dce (DEFAULT_TABLE table=default.Table_a55db7a900494ec8bb877fd245f39f60 [id=a36083ca5811443bbc0a73e3d7778c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.488435 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa06277c743464dad2ae9a396ae7dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.489248 1165 ts_tablet_manager.cc:1918] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:05.489362 1165 log.cc:1199] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/b1acc41a93f6433588727811903921fe
I20250625 13:52:05.489794 1165 ts_tablet_manager.cc:1939] T b1acc41a93f6433588727811903921fe P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.491572 1165 tablet_replica.cc:331] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.491845 1165 raft_consensus.cc:2241] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.492064 1165 raft_consensus.cc:2270] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.493721 1842 raft_consensus.cc:491] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.493902 1842 raft_consensus.cc:513] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.494014 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9f24ae7c08bd42959ab48f7007ec98d3 (DEFAULT_TABLE table=default.Table_61c4552a00dd4dfb882f0e86f3b8776a [id=5d940360188f49d9b9e4f7b737a163c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.494226 887 tablet_service.cc:1468] Processing CreateTablet for tablet 3fc555eb21fd469185828c3670006daf (DEFAULT_TABLE table=default.Table_fe7c9a0a684b4277a14d8ed100f3fa74 [id=33bfa70f7d944ff1b7e808288266b1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.494401 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f24ae7c08bd42959ab48f7007ec98d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.494647 1842 leader_election.cc:290] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:05.495179 881 tablet_service.cc:1468] Processing CreateTablet for tablet 6e2d109e6fce407dad72a2469479b5ed (DEFAULT_TABLE table=default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98 [id=e06a5b87bfb44b23beaeee29abf42788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.495398 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 991bb428def545fd979e9f1a80ce4d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.495618 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b7062c34fd143bf86b743055a27db8c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.496441 849 leader_election.cc:343] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:05.496811 851 leader_election.cc:343] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8b7062c34fd143bf86b743055a27db8c
I20250625 13:52:05.496950 851 leader_election.cc:304] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.494649 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc555eb21fd469185828c3670006daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.497895 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2d109e6fce407dad72a2469479b5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.498276 1842 raft_consensus.cc:2747] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.498524 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet c0a3ea18bab944749d45da9b3139f8bc (table default.Table_c8b06a7351fb437aa0440f6fa3898939 [id=eac16bb1f501401c9f6803136dc04116]) successfully deleted
W20250625 13:52:05.498798 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 90723.5us
W20250625 13:52:05.499111 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 81a359c798024b2a99aa3b307cc1967f (table default.Table_da4c3565bbf845ca9a26fe3d1f39f99f [id=ebbdfe16e6f64ccd95e6dc7d068cd814], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 81a359c798024b2a99aa3b307cc1967f consensus state has no leader
W20250625 13:52:05.501592 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 58a532e889244b088562081e444234a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58a532e889244b088562081e444234a3 already in progress: creating tablet
I20250625 13:52:05.501868 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]) successfully deleted
W20250625 13:52:05.502260 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet da2f8d0d9a714a28ba6fb9642d461de2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da2f8d0d9a714a28ba6fb9642d461de2 already in progress: creating tablet
W20250625 13:52:05.502460 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 00d4acd387c84d55834f632f41045862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00d4acd387c84d55834f632f41045862 already in progress: creating tablet
W20250625 13:52:05.502985 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c4db530b54f04fedb18bf993dc60d8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4db530b54f04fedb18bf993dc60d8da already in progress: creating tablet
I20250625 13:52:05.506538 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3fc555eb21fd469185828c3670006daf (DEFAULT_TABLE table=default.Table_fe7c9a0a684b4277a14d8ed100f3fa74 [id=33bfa70f7d944ff1b7e808288266b1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.506637 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 6e2d109e6fce407dad72a2469479b5ed (DEFAULT_TABLE table=default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98 [id=e06a5b87bfb44b23beaeee29abf42788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.506814 1170 tablet_bootstrap.cc:654] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.506938 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc555eb21fd469185828c3670006daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.510030 592 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8304ee2e3dce49049c2320b2ba363485
I20250625 13:52:05.510741 1165 ts_tablet_manager.cc:1905] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.512182 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20227b2471d345dc90782795a2d1de32" } new_table_name: "default.Table_ac6acb4e2c0e4001927ffb4230ad2ecf" modify_external_catalogs: true
I20250625 13:52:05.514950 1168 ts_tablet_manager.cc:1918] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:05.515170 1168 log.cc:1199] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d4f19ddc51324cc18242c6081f3f7de5
I20250625 13:52:05.515681 1168 ts_tablet_manager.cc:1939] T d4f19ddc51324cc18242c6081f3f7de5 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.516158 1171 tablet_bootstrap.cc:492] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.516325 1171 ts_tablet_manager.cc:1397] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.010s
I20250625 13:52:05.516656 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 397) took 6494 ms. Trace:
I20250625 13:52:05.516819 1168 rpcz_store.cc:276] 0625 13:51:59.022501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.022568 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:05.436198 (+6413630us) ts_tablet_manager.cc:1145] Deleting tablet d4f19ddc51324cc18242c6081f3f7de5
0625 13:52:05.436208 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.490122 (+ 53914us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.514807 (+ 24685us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.516624 (+ 1817us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6413355}
I20250625 13:52:05.517330 1168 tablet_replica.cc:331] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.517588 1168 raft_consensus.cc:2241] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.517771 1168 pending_rounds.cc:70] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:05.517771 1171 raft_consensus.cc:357] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.517936 1168 pending_rounds.cc:77] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520592048504832 op_type: NO_OP noop_request { }
I20250625 13:52:05.518121 1168 raft_consensus.cc:2887] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:05.518260 1168 raft_consensus.cc:2270] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.518784 1166 tablet_bootstrap.cc:492] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.518785 1168 ts_tablet_manager.cc:1905] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.518965 1166 ts_tablet_manager.cc:1397] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.014s
I20250625 13:52:05.519923 1166 raft_consensus.cc:357] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.520172 1166 raft_consensus.cc:383] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.520244 1166 raft_consensus.cc:738] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.520824 1166 consensus_queue.cc:260] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.522588 1165 ts_tablet_manager.cc:1918] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.522722 1165 log.cc:1199] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/fd39216b01054ff38ec9175d7a2f87f2
I20250625 13:52:05.523947 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 54fa042947da4e1b89018640b46b271a with delete_type TABLET_DATA_DELETED (replaced by tablet ab7202a8761444a0a731993d890fcdeb) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.524298 1166 ts_tablet_manager.cc:1428] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:52:05.524287 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d4acd387c84d55834f632f41045862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.518021 1171 raft_consensus.cc:383] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.524467 1171 raft_consensus.cc:738] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.524551 887 tablet_service.cc:1515] Processing DeleteTablet for tablet c4db530b54f04fedb18bf993dc60d8da with delete_type TABLET_DATA_DELETED (replaced by tablet c366da4fb138453386d97fba0f5c91be) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.524722 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9c1deb52366c404cbdc08076d4bcb3d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c1deb52366c404cbdc08076d4bcb3d1 already in progress: creating tablet
I20250625 13:52:05.524827 888 tablet_service.cc:1515] Processing DeleteTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.524988 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet fb4973cbffc743dba98d7b7c9f76be3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb4973cbffc743dba98d7b7c9f76be3f already in progress: creating tablet
I20250625 13:52:05.525017 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 58a532e889244b088562081e444234a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c4a6d7d0c9647a4a17dca28d6a063a5) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.525199 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 69672579a83c41d09ee7c78a6fa7d746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69672579a83c41d09ee7c78a6fa7d746 already in progress: creating tablet
W20250625 13:52:05.525381 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d4fa8abea8d4eee8651be9bf4b2b34f already in progress: creating tablet
W20250625 13:52:05.525595 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8063122ea7b24b6790f5c3cc884e09be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8063122ea7b24b6790f5c3cc884e09be already in progress: creating tablet
I20250625 13:52:05.525676 1171 consensus_queue.cc:260] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.525768 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd]) successfully deleted
I20250625 13:52:05.525873 881 tablet_service.cc:1515] Processing DeleteTablet for tablet b1acc41a93f6433588727811903921fe with delete_type TABLET_DATA_DELETED (replaced by tablet d5cb3b35a5c2483fa82226133a7c161d) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.525992 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 57afeff143c846289c6159974b00d7ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57afeff143c846289c6159974b00d7ba already in progress: creating tablet
I20250625 13:52:05.526473 1166 tablet_bootstrap.cc:492] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:05.529278 851 consensus_peers.cc:487] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1acc41a93f6433588727811903921fe. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:05.531256 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 19017cc372af43b2b5074a8990375ede with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.532049 1171 ts_tablet_manager.cc:1428] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.016s user 0.001s sys 0.002s
I20250625 13:52:05.532392 1171 tablet_bootstrap.cc:492] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.528016 1166 tablet_bootstrap.cc:654] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:05.536659 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a89f9ffb5e4643f8915e9fd69dd3e842 consensus state has no leader
I20250625 13:52:05.535270 593 master_service.cc:948] ReplaceTablet: received request to replace tablet d8985f259eb54ffdab29a9f6a2c5ae16 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.541024 1166 tablet_bootstrap.cc:492] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.541153 1166 ts_tablet_manager.cc:1397] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250625 13:52:05.542825 1166 raft_consensus.cc:357] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.542979 1166 raft_consensus.cc:383] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.543018 1166 raft_consensus.cc:738] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.543218 1166 consensus_queue.cc:260] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.544648 1166 ts_tablet_manager.cc:1428] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:05.545590 1166 tablet_bootstrap.cc:492] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.547272 1166 tablet_bootstrap.cc:654] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.551901 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2d109e6fce407dad72a2469479b5ed. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:05.554195 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a6ff3b675aba42adb035ff5d1404a889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6ff3b675aba42adb035ff5d1404a889 already in progress: creating tablet
W20250625 13:52:05.554684 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 60ef50a3624b4c79acc803ff0e2ed364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ef50a3624b4c79acc803ff0e2ed364 already in progress: creating tablet
W20250625 13:52:05.555006 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 39540b81fd4746858e65be49ded0e1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39540b81fd4746858e65be49ded0e1df already in progress: creating tablet
W20250625 13:52:05.555174 849 consensus_peers.cc:487] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf1664afbab647b3baeed6391196bed9. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:05.555392 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2c8cf5d6cd624a1797486c69bd2098fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8cf5d6cd624a1797486c69bd2098fd already in progress: creating tablet
W20250625 13:52:05.555650 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d2f545a5934748b2af767d37ea937fca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2f545a5934748b2af767d37ea937fca already in progress: creating tablet
I20250625 13:52:05.557612 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.558810 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 58a532e889244b088562081e444234a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c4a6d7d0c9647a4a17dca28d6a063a5) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.559023 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 19017cc372af43b2b5074a8990375ede with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.559175 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c4db530b54f04fedb18bf993dc60d8da with delete_type TABLET_DATA_DELETED (replaced by tablet c366da4fb138453386d97fba0f5c91be) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.559345 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d4acd387c84d55834f632f41045862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:05.567284 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3016763b9e294b9da2861fe97e57e241 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3016763b9e294b9da2861fe97e57e241 already in progress: creating tablet
W20250625 13:52:05.567759 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 93fdde7d068b400ba7070166eff63502 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93fdde7d068b400ba7070166eff63502 already in progress: creating tablet
W20250625 13:52:05.568048 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f197ea67c34546a7981bcd9df94702e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f197ea67c34546a7981bcd9df94702e6 already in progress: creating tablet
W20250625 13:52:05.568293 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c0f461a7a7ef45009279716a3024ee26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f461a7a7ef45009279716a3024ee26 already in progress: creating tablet
W20250625 13:52:05.574640 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0e77736191544e8da32ca9927f0a5804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e77736191544e8da32ca9927f0a5804 already in progress: creating tablet
I20250625 13:52:05.574994 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 04479dfb34584f0891c604ef72bf93b5 (table default.Table_c1e172984ca643a9b7c88d09f4d5221a [id=d0ac91ff8e7f473b8b2f938243067eac]) successfully deleted
I20250625 13:52:05.575222 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 7d361e73fe3d4ea2bb6735f814345ebb (table default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]) successfully deleted
W20250625 13:52:05.575466 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3c6dd5b87ff34151836a3c29740e7c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c6dd5b87ff34151836a3c29740e7c11 already in progress: creating tablet
W20250625 13:52:05.575726 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7db450dc63bd4b6caf074c67490cd175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db450dc63bd4b6caf074c67490cd175 already in progress: creating tablet
W20250625 13:52:05.575969 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 455884b62ee44b6da53543093f721027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 455884b62ee44b6da53543093f721027 already in progress: creating tablet
W20250625 13:52:05.576206 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet be3313ec5dfb41b68662d3ff4c8c94f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be3313ec5dfb41b68662d3ff4c8c94f9 already in progress: creating tablet
W20250625 13:52:05.576409 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4d25e4114a184bc99705042593ab4ab9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d25e4114a184bc99705042593ab4ab9 already in progress: creating tablet
I20250625 13:52:05.576617 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet cd4f0857f52449e1b26fde09c6c7c9d4 (table default.Table_fdde488d5ee94a8593bf9fc6205adeac [id=a2fabddfeb834ef68d234cd878a338c9]) successfully deleted
W20250625 13:52:05.576854 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e3b3d5a2ab7042c2a2f18890363e7a9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b3d5a2ab7042c2a2f18890363e7a9b already in progress: creating tablet
W20250625 13:52:05.577111 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d7660a2b70744bd589220b28e6edbc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7660a2b70744bd589220b28e6edbc42 already in progress: creating tablet
W20250625 13:52:05.577505 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eabd67b9ceb74bd1a5f4f115c7db0e87 already in progress: creating tablet
W20250625 13:52:05.583516 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c737daaede054d09a0c07564157423a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c737daaede054d09a0c07564157423a9 already in progress: creating tablet
W20250625 13:52:05.583819 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet eafc402d809245d189f3977162506ef8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eafc402d809245d189f3977162506ef8 already in progress: creating tablet
I20250625 13:52:05.583959 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_feec880c1f294ed2a18e041a5bc7a405"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.587531 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feec880c1f294ed2a18e041a5bc7a405 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.584379 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d0ce3bb2d8b04255ae3bc517960db379"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.588115 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet b1acc41a93f6433588727811903921fe with delete_type TABLET_DATA_DELETED (replaced by tablet d5cb3b35a5c2483fa82226133a7c161d) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:05.588379 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0ce3bb2d8b04255ae3bc517960db379 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:05.589077 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 067272869c5547648c90f83a34a036aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 067272869c5547648c90f83a34a036aa already in progress: creating tablet
I20250625 13:52:05.584776 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3947dfd6b9454b7eb988acee97d6e311" } modify_external_catalogs: true
I20250625 13:52:05.593904 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3947dfd6b9454b7eb988acee97d6e311" } modify_external_catalogs: true
W20250625 13:52:05.594518 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8ebe47e363ce4a6c8129169bc6777e63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ebe47e363ce4a6c8129169bc6777e63 already in progress: creating tablet
W20250625 13:52:05.594954 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e8d2ab8f91a045a3a39f3d09459f2a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8d2ab8f91a045a3a39f3d09459f2a28 already in progress: creating tablet
I20250625 13:52:05.595197 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet bb1c7a85da844d34be805843bc60d350 (table default.Table_c6d92fec1b3f4f368c408fc302731df0 [id=2d944b43ce6048008e959fcf6747eaae]) successfully deleted
I20250625 13:52:05.595585 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 643928ca4fc94ce58431a0525f2abb99 (table default.Table_ad4e07371e3b42c5b05954f9537fd6b7 [id=c861ee66493e47ec97ad3aa9a6495499]) successfully deleted
I20250625 13:52:05.598306 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.598856 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 8149aea0ff7646d3b94c9159f56e48ef (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]) successfully deleted
I20250625 13:52:05.599331 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet b7641869a0bb463698244ff445d87a1f (table default.Table_6e5d1377cfc84651afd53a78735468d8 [id=bd476a97393841899b4cdb426c39e6e9]) successfully deleted
I20250625 13:52:05.605108 1165 ts_tablet_manager.cc:1939] T fd39216b01054ff38ec9175d7a2f87f2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:52:05.605754 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ccce48011a15405db2b95c04c9b01b3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccce48011a15405db2b95c04c9b01b3c already in progress: creating tablet
I20250625 13:52:05.606141 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet fd39216b01054ff38ec9175d7a2f87f2 (table default.Table_c54f67fabc114fc6bb555987a103adba [id=aa481ec0dc6e42b391641f4c3d05ca42]) successfully deleted
I20250625 13:52:05.606711 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet d4f19ddc51324cc18242c6081f3f7de5 (table default.Table_08c36a3d2a03417982751548b18def03 [id=b8285b64bbee49f9be8741b252d03363]) successfully deleted
I20250625 13:52:05.610939 1171 tablet_bootstrap.cc:654] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.611881 1165 tablet_replica.cc:331] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.612069 1165 raft_consensus.cc:2241] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.612212 1165 raft_consensus.cc:2270] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.613538 1165 ts_tablet_manager.cc:1905] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:05.615222 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f56a8a6cb964d3a979e2f1d944eeddc consensus state has no leader
W20250625 13:52:05.619788 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68a7712455694a1cbead413230620827 consensus state has no leader
I20250625 13:52:05.621469 1168 ts_tablet_manager.cc:1918] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.621573 1168 log.cc:1199] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/583ee2f2c66841939c93104efc36facb
I20250625 13:52:05.621924 1168 ts_tablet_manager.cc:1939] T 583ee2f2c66841939c93104efc36facb P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.622417 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.622566 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:05.622720 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 398) took 6559 ms. Trace:
I20250625 13:52:05.622789 1168 rpcz_store.cc:276] 0625 13:51:59.063749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.063802 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:05.517261 (+6453459us) ts_tablet_manager.cc:1145] Deleting tablet 583ee2f2c66841939c93104efc36facb
0625 13:52:05.517271 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.581544 (+ 64273us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.621361 (+ 39817us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.622700 (+ 1339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6453217}
I20250625 13:52:05.623842 1168 tablet_replica.cc:331] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.624047 1168 raft_consensus.cc:2241] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.624184 1168 raft_consensus.cc:2270] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.624670 1168 ts_tablet_manager.cc:1905] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:05.625715 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b0e49ca4dee42c59c0a2927a47aa320 consensus state has no leader
W20250625 13:52:05.625689 592 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 6d95a376304e464ab75e2e1e00e7a924 consensus state has no leader
I20250625 13:52:05.626667 1710 tablet_replica.cc:331] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.626838 1710 raft_consensus.cc:2241] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.626965 1710 raft_consensus.cc:2270] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.627293 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:05.627424 1710 ts_tablet_manager.cc:1905] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.628262 595 catalog_manager.cc:5582] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:05.628643 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
I20250625 13:52:05.629107 1171 tablet_bootstrap.cc:492] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.629212 1171 ts_tablet_manager.cc:1397] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.097s user 0.016s sys 0.003s
I20250625 13:52:05.629642 595 catalog_manager.cc:5582] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:05.630082 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_2dbd2e7b0a624bcb99f8edc7918b558d [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 2 got 0
I20250625 13:52:05.631083 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3b8e232f80724e38992aa5d66fd0caf3 (DEFAULT_TABLE table=default.Table_70a5975b997b4664ae947803f73e1a27 [id=6c81574e23834914a987a9c4003de66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.631155 1171 raft_consensus.cc:357] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.631327 1171 raft_consensus.cc:383] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.631337 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 583ee2f2c66841939c93104efc36facb (table default.Table_5964baca5b4a4701bf5a1b86d494b4e0 [id=c12c680f4ece4270ba65d0e2dc275254]) successfully deleted
I20250625 13:52:05.631389 1171 raft_consensus.cc:738] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.631526 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e232f80724e38992aa5d66fd0caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.631700 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet f033bfa4925b4d1c971789eb77efd92c (table default.Table_df8e6bfa3a4741fca89b09188c645be9 [id=e8ccbd9e2b83408b87689d6d6f26eb47]) successfully deleted
I20250625 13:52:05.632382 1171 consensus_queue.cc:260] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.632802 1171 ts_tablet_manager.cc:1428] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:05.633051 1171 tablet_bootstrap.cc:492] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.636168 884 tablet_service.cc:1468] Processing CreateTablet for tablet 34b61a29713e4183ac17e7414807770f (DEFAULT_TABLE table=default.Table_30935873709947f3adc8b29d8ffc698c [id=e029c1956adc49c1a6c1e377e3ef7628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.636499 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b61a29713e4183ac17e7414807770f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.636442 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3b8e232f80724e38992aa5d66fd0caf3 (DEFAULT_TABLE table=default.Table_70a5975b997b4664ae947803f73e1a27 [id=6c81574e23834914a987a9c4003de66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.636710 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e232f80724e38992aa5d66fd0caf3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:05.638495 849 consensus_peers.cc:487] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e0f2fa5d5614ad394e6e12ec62814d4. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:05.639463 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 27092991fcad475fabc77d1f8bab0582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.639714 891 tablet_service.cc:1468] Processing CreateTablet for tablet 8416186aa9f64c1ca4a215cedd70f507 (DEFAULT_TABLE table=default.Table_cd9897451d5b458bb4135bc901f46208 [id=eff4e0c315354603b4932490fc665cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.639849 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3b8e232f80724e38992aa5d66fd0caf3 (DEFAULT_TABLE table=default.Table_70a5975b997b4664ae947803f73e1a27 [id=6c81574e23834914a987a9c4003de66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.640007 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8416186aa9f64c1ca4a215cedd70f507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.640262 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e232f80724e38992aa5d66fd0caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.641767 1165 ts_tablet_manager.cc:1918] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.641897 1165 log.cc:1199] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/abd6f9121a9d4b39adde1b880a875e23
I20250625 13:52:05.642176 1165 ts_tablet_manager.cc:1939] T abd6f9121a9d4b39adde1b880a875e23 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.643050 1165 tablet_replica.cc:331] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.643224 1165 raft_consensus.cc:2241] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.643180 881 tablet_service.cc:1468] Processing CreateTablet for tablet 6c183b3ba6f145e4b4741008ca27588a (DEFAULT_TABLE table=default.Table_4017f3e1f1534574b8241c6d417fec78 [id=2c706b062d2c4eb7ad30242b4b177a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.643694 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 8416186aa9f64c1ca4a215cedd70f507 (DEFAULT_TABLE table=default.Table_cd9897451d5b458bb4135bc901f46208 [id=eff4e0c315354603b4932490fc665cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.644021 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8416186aa9f64c1ca4a215cedd70f507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.647869 892 tablet_service.cc:1468] Processing CreateTablet for tablet bdc5395c1de349b9b20d28625b8751f7 (DEFAULT_TABLE table=default.Table_997801fa9593458d9642268646093de6 [id=1aa7ebee65004b51bdf513cbe73c4a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.648046 1171 tablet_bootstrap.cc:654] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.648207 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc5395c1de349b9b20d28625b8751f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.648442 1026 tablet_service.cc:1468] Processing CreateTablet for tablet bdc5395c1de349b9b20d28625b8751f7 (DEFAULT_TABLE table=default.Table_997801fa9593458d9642268646093de6 [id=1aa7ebee65004b51bdf513cbe73c4a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.648784 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc5395c1de349b9b20d28625b8751f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.648783 888 tablet_service.cc:1468] Processing CreateTablet for tablet 1c6ec3a7578e4892b6193044c58b142e (DEFAULT_TABLE table=default.Table_2733881e50814fb49b1cb5ee2e42f772 [id=9f8095c74cee4380a5a5ab30f83ceaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.649914 741 tablet_service.cc:1468] Processing CreateTablet for tablet 34b61a29713e4183ac17e7414807770f (DEFAULT_TABLE table=default.Table_30935873709947f3adc8b29d8ffc698c [id=e029c1956adc49c1a6c1e377e3ef7628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.650285 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b61a29713e4183ac17e7414807770f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.650458 1171 tablet_bootstrap.cc:492] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.650570 1171 ts_tablet_manager.cc:1397] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250625 13:52:05.651577 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 1c6ec3a7578e4892b6193044c58b142e (DEFAULT_TABLE table=default.Table_2733881e50814fb49b1cb5ee2e42f772 [id=9f8095c74cee4380a5a5ab30f83ceaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.651655 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6c183b3ba6f145e4b4741008ca27588a (DEFAULT_TABLE table=default.Table_4017f3e1f1534574b8241c6d417fec78 [id=2c706b062d2c4eb7ad30242b4b177a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.651880 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6ec3a7578e4892b6193044c58b142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.652046 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c183b3ba6f145e4b4741008ca27588a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.652377 1165 raft_consensus.cc:2270] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.652884 1165 ts_tablet_manager.cc:1905] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.655501 743 tablet_service.cc:1468] Processing CreateTablet for tablet 6c183b3ba6f145e4b4741008ca27588a (DEFAULT_TABLE table=default.Table_4017f3e1f1534574b8241c6d417fec78 [id=2c706b062d2c4eb7ad30242b4b177a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.655622 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 34b61a29713e4183ac17e7414807770f (DEFAULT_TABLE table=default.Table_30935873709947f3adc8b29d8ffc698c [id=e029c1956adc49c1a6c1e377e3ef7628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.651222 1171 raft_consensus.cc:357] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.655799 1171 raft_consensus.cc:383] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.655830 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c183b3ba6f145e4b4741008ca27588a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.655872 1171 raft_consensus.cc:738] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.655946 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b61a29713e4183ac17e7414807770f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.659593 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2385e066be614e78b1ae954685f86354 (DEFAULT_TABLE table=default.Table_8fb483ed4e0140d8ba66be83ccbe58b6 [id=c7168cb0a6634957b16f484bd6471dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.659605 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 2385e066be614e78b1ae954685f86354 (DEFAULT_TABLE table=default.Table_8fb483ed4e0140d8ba66be83ccbe58b6 [id=c7168cb0a6634957b16f484bd6471dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.659641 1171 consensus_queue.cc:260] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.659891 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2385e066be614e78b1ae954685f86354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.659900 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2385e066be614e78b1ae954685f86354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.660960 1171 ts_tablet_manager.cc:1428] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 13:52:05.661224 1171 tablet_bootstrap.cc:492] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.661224 887 tablet_service.cc:1468] Processing CreateTablet for tablet 614181c6e4d24e7b9e5da3b4818cf065 (DEFAULT_TABLE table=default.Table_a637d32e43d647768eebfd52dd249a2c [id=d9144cc898094039a4d89e8d58f1ae86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.661751 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 614181c6e4d24e7b9e5da3b4818cf065 (DEFAULT_TABLE table=default.Table_a637d32e43d647768eebfd52dd249a2c [id=d9144cc898094039a4d89e8d58f1ae86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.661901 893 tablet_service.cc:1468] Processing CreateTablet for tablet 07cfdcf3e8b74c24b48b62bdc8bde971 (DEFAULT_TABLE table=default.Table_3ade196aa4334cff912b6e96e532483c [id=77b0be57f43c4c628b0b9338a7fe77ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.662051 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 614181c6e4d24e7b9e5da3b4818cf065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.662518 1171 tablet_bootstrap.cc:654] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.662485 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 07cfdcf3e8b74c24b48b62bdc8bde971 (DEFAULT_TABLE table=default.Table_3ade196aa4334cff912b6e96e532483c [id=77b0be57f43c4c628b0b9338a7fe77ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.662765 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cfdcf3e8b74c24b48b62bdc8bde971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.663547 752 tablet_service.cc:1468] Processing CreateTablet for tablet bdc5395c1de349b9b20d28625b8751f7 (DEFAULT_TABLE table=default.Table_997801fa9593458d9642268646093de6 [id=1aa7ebee65004b51bdf513cbe73c4a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.663846 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdc5395c1de349b9b20d28625b8751f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.667800 751 tablet_service.cc:1468] Processing CreateTablet for tablet 8416186aa9f64c1ca4a215cedd70f507 (DEFAULT_TABLE table=default.Table_cd9897451d5b458bb4135bc901f46208 [id=eff4e0c315354603b4932490fc665cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.668092 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8416186aa9f64c1ca4a215cedd70f507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.672336 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet abd6f9121a9d4b39adde1b880a875e23 (table default.Table_5d5a62a1608c45929c8fa1c853a06a27 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]) successfully deleted
I20250625 13:52:05.672696 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c183b3ba6f145e4b4741008ca27588a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.675752 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 614181c6e4d24e7b9e5da3b4818cf065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.679004 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fbe9775e7dbd44ce9fd173057e030bd7" } new_table_name: "default.Table_ed893da9dad747c7869db417fd1c55e6" modify_external_catalogs: true
I20250625 13:52:05.680331 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6ec3a7578e4892b6193044c58b142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.680944 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cfdcf3e8b74c24b48b62bdc8bde971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.684336 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2385e066be614e78b1ae954685f86354 (DEFAULT_TABLE table=default.Table_8fb483ed4e0140d8ba66be83ccbe58b6 [id=c7168cb0a6634957b16f484bd6471dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.685871 1165 ts_tablet_manager.cc:1918] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.686004 1165 log.cc:1199] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/f065b4e3218b4e6e8a2ac3a70eede13b
I20250625 13:52:05.686280 1165 ts_tablet_manager.cc:1939] T f065b4e3218b4e6e8a2ac3a70eede13b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.688048 1166 tablet_bootstrap.cc:492] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.688145 757 tablet_service.cc:1468] Processing CreateTablet for tablet 1c6ec3a7578e4892b6193044c58b142e (DEFAULT_TABLE table=default.Table_2733881e50814fb49b1cb5ee2e42f772 [id=9f8095c74cee4380a5a5ab30f83ceaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.688499 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6ec3a7578e4892b6193044c58b142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.689543 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:05.689725 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dc75a0a8421b457089cc707d9eb6b76f (table default.Table_034f6cf597cd469fba12687016eff7d1 [id=fa0246aff3f648538e6fddcb30f3a399]). Expected version 2 got 0
I20250625 13:52:05.691028 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_194c4cbe54834768a869225974bea8ab" } modify_external_catalogs: true
I20250625 13:52:05.691200 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_194c4cbe54834768a869225974bea8ab" } modify_external_catalogs: true
I20250625 13:52:05.691589 746 tablet_service.cc:1468] Processing CreateTablet for tablet 07cfdcf3e8b74c24b48b62bdc8bde971 (DEFAULT_TABLE table=default.Table_3ade196aa4334cff912b6e96e532483c [id=77b0be57f43c4c628b0b9338a7fe77ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.691799 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1f52a2da8b374d99952264989d3f5b90" } modify_external_catalogs: true
I20250625 13:52:05.691967 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cfdcf3e8b74c24b48b62bdc8bde971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.691977 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1f52a2da8b374d99952264989d3f5b90" } modify_external_catalogs: true
I20250625 13:52:05.693269 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4fcd474fe7fd48c0b7a71988682079c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.693768 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fcd474fe7fd48c0b7a71988682079c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.695235 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4cf923b8b24141eeab3c245e08664637" } modify_external_catalogs: true
I20250625 13:52:05.695446 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4cf923b8b24141eeab3c245e08664637" } modify_external_catalogs: true
I20250625 13:52:05.688102 739 tablet_service.cc:1468] Processing CreateTablet for tablet 614181c6e4d24e7b9e5da3b4818cf065 (DEFAULT_TABLE table=default.Table_a637d32e43d647768eebfd52dd249a2c [id=d9144cc898094039a4d89e8d58f1ae86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:05.688180 1166 ts_tablet_manager.cc:1397] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.143s user 0.014s sys 0.009s
I20250625 13:52:05.695933 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 614181c6e4d24e7b9e5da3b4818cf065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.696432 1166 raft_consensus.cc:357] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.696583 1166 raft_consensus.cc:383] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.696708 1166 raft_consensus.cc:738] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.696916 1166 consensus_queue.cc:260] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.696763 1748 raft_consensus.cc:491] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.698987 1748 raft_consensus.cc:513] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.679296 1710 ts_tablet_manager.cc:1918] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.699234 1710 log.cc:1199] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/055f88e118454e32baa578cae9aceb0c
I20250625 13:52:05.699563 1710 ts_tablet_manager.cc:1939] T 055f88e118454e32baa578cae9aceb0c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.699662 1748 leader_election.cc:290] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:05.699566 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_945803b681074611b2bfcd213a5e072d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.700126 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_945803b681074611b2bfcd213a5e072d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.700232 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e29e7b55570483b80c6e7a1f227fa7b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:05.680740 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 01ba67b500b646d7aa55315f7ee23ed6 with delete_type TABLET_DATA_DELETED (replaced by tablet 32143e2497ee4d1b8c2e7cbfc6160ef4) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.693653 1170 tablet_bootstrap.cc:492] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:05.700608 1170 ts_tablet_manager.cc:1397] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.216s user 0.007s sys 0.007s
W20250625 13:52:05.700732 849 leader_election.cc:343] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.701220 1166 ts_tablet_manager.cc:1428] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:05.701395 1170 raft_consensus.cc:357] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.701541 1170 raft_consensus.cc:383] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.702032 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2385e066be614e78b1ae954685f86354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:05.702733 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 055f88e118454e32baa578cae9aceb0c (table default.Table_df6a42eba40144cf94d09e54694a85b9 [id=e50f8ace632e4ea284d17b85f474cef5]) successfully deleted
I20250625 13:52:05.701597 1170 raft_consensus.cc:738] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:05.703631 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 254afffd283449fdaf5e190ba3028f16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 254afffd283449fdaf5e190ba3028f16 already in progress: creating tablet
I20250625 13:52:05.703671 1170 consensus_queue.cc:260] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:05.704190 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 already in progress: creating tablet
W20250625 13:52:05.704762 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b7d888e628a94f939ecf2571e60b1948 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d888e628a94f939ecf2571e60b1948 already in progress: creating tablet
I20250625 13:52:05.704860 1166 tablet_bootstrap.cc:492] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.705039 1710 tablet_replica.cc:331] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.705209 1710 raft_consensus.cc:2241] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.706039 1171 tablet_bootstrap.cc:492] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.706226 1171 ts_tablet_manager.cc:1397] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.002s
I20250625 13:52:05.706905 1171 raft_consensus.cc:357] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.707119 1171 raft_consensus.cc:383] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.707206 1171 raft_consensus.cc:738] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.707413 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 455884b62ee44b6da53543093f721027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.707499 1171 consensus_queue.cc:260] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.707748 1165 tablet_replica.cc:331] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.707942 1165 raft_consensus.cc:2241] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.708124 1165 raft_consensus.cc:2270] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.708803 1171 ts_tablet_manager.cc:1428] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:05.709334 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet f065b4e3218b4e6e8a2ac3a70eede13b (table default.Table_71166441017840fa9a67752209689ee2 [id=fc9bcf7791ec4c069f0bcae18943cac9]) successfully deleted
I20250625 13:52:05.710420 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 60ef50a3624b4c79acc803ff0e2ed364 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.711074 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 with delete_type TABLET_DATA_DELETED (replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.711212 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.714766 1710 pending_rounds.cc:70] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:05.715248 1710 pending_rounds.cc:77] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520614499442688 op_type: NO_OP noop_request { }
I20250625 13:52:05.716063 1165 ts_tablet_manager.cc:1905] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.716027 1710 raft_consensus.cc:2887] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:05.716979 1170 ts_tablet_manager.cc:1428] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
W20250625 13:52:05.717130 851 leader_election.cc:343] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8e29e7b55570483b80c6e7a1f227fa7b
I20250625 13:52:05.717245 851 leader_election.cc:304] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.716897 1710 raft_consensus.cc:2270] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.717626 1748 raft_consensus.cc:2747] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.718881 1171 tablet_bootstrap.cc:492] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.719509 1170 tablet_bootstrap.cc:492] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:05.721715 1171 tablet_bootstrap.cc:654] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.732564 1165 ts_tablet_manager.cc:1918] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.732697 1165 log.cc:1199] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/455884b62ee44b6da53543093f721027
I20250625 13:52:05.733006 1165 ts_tablet_manager.cc:1939] T 455884b62ee44b6da53543093f721027 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.738093 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d8985f259eb54ffdab29a9f6a2c5ae16
I20250625 13:52:05.738690 593 catalog_manager.cc:6652] ReplaceTablet: tablet d8985f259eb54ffdab29a9f6a2c5ae16 deleted and replaced by tablet 879d91dcfb0b4868a7ba6833c211e5eb
I20250625 13:52:05.739327 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 with delete_type TABLET_DATA_DELETED (replaced by tablet 879d91dcfb0b4868a7ba6833c211e5eb) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.740465 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cab4329d58404014833ae5029f8f296d
I20250625 13:52:05.741479 745 tablet_service.cc:1515] Processing DeleteTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 with delete_type TABLET_DATA_DELETED (replaced by tablet 879d91dcfb0b4868a7ba6833c211e5eb) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.741899 740 tablet_service.cc:1515] Processing DeleteTablet for tablet cab4329d58404014833ae5029f8f296d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.742864 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet cab4329d58404014833ae5029f8f296d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.747716 1170 tablet_bootstrap.cc:654] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.754976 1166 tablet_bootstrap.cc:654] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.756978 1748 raft_consensus.cc:491] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.757157 1748 raft_consensus.cc:513] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.757807 1748 leader_election.cc:290] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:05.758522 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c666ff2d6b9b4ed4a18e370826c143fc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.758852 851 leader_election.cc:343] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: c666ff2d6b9b4ed4a18e370826c143fc
W20250625 13:52:05.759300 849 leader_election.cc:343] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.759441 849 leader_election.cc:304] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.759881 1748 raft_consensus.cc:2747] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.759881 1171 tablet_bootstrap.cc:492] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.760259 1171 ts_tablet_manager.cc:1397] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.000s
I20250625 13:52:05.760962 1171 raft_consensus.cc:357] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.761188 1171 raft_consensus.cc:383] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.761276 1171 raft_consensus.cc:738] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.761521 1171 consensus_queue.cc:260] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.762907 1171 ts_tablet_manager.cc:1428] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 13:52:05.763935 1171 tablet_bootstrap.cc:492] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.764091 892 tablet_service.cc:1515] Processing DeleteTablet for tablet cab4329d58404014833ae5029f8f296d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.764257 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 455884b62ee44b6da53543093f721027 (table default.Table_f51c5c5fe5394b598404b1944ee7d0b5 [id=67c521d398af4fc4bfe9e6c23d400906]) successfully deleted
I20250625 13:52:05.764318 881 tablet_service.cc:1515] Processing DeleteTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 with delete_type TABLET_DATA_DELETED (replaced by tablet 879d91dcfb0b4868a7ba6833c211e5eb) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.764514 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d8985f259eb54ffdab29a9f6a2c5ae16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8985f259eb54ffdab29a9f6a2c5ae16 already in progress: creating tablet
W20250625 13:52:05.764780 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet cab4329d58404014833ae5029f8f296d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cab4329d58404014833ae5029f8f296d already in progress: creating tablet
I20250625 13:52:05.765072 1710 ts_tablet_manager.cc:1905] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.765410 1171 tablet_bootstrap.cc:654] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.778388 1168 ts_tablet_manager.cc:1918] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.778527 1168 log.cc:1199] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/34ab663b7eba452c9f9775c0f568010b
I20250625 13:52:05.778875 1168 ts_tablet_manager.cc:1939] T 34ab663b7eba452c9f9775c0f568010b P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.779672 1171 tablet_bootstrap.cc:492] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.780222 1171 ts_tablet_manager.cc:1397] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250625 13:52:05.781116 1171 raft_consensus.cc:357] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.782076 1171 raft_consensus.cc:383] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.782231 1171 raft_consensus.cc:738] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.782610 1171 consensus_queue.cc:260] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.784734 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9d8551fdf4e043cd90e2ce8228c3db7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.785558 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdca99b734448118e0c42227c78e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.785856 1165 tablet_replica.cc:331] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.786060 1165 raft_consensus.cc:2241] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.786218 1165 raft_consensus.cc:2270] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.786296 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_da32ce790d1e43469d2ece95ae454446" } modify_external_catalogs: true
I20250625 13:52:05.786494 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_da32ce790d1e43469d2ece95ae454446" } modify_external_catalogs: true
I20250625 13:52:05.786984 1165 ts_tablet_manager.cc:1905] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.788709 1171 ts_tablet_manager.cc:1428] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250625 13:52:05.788980 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4a105e80c1460cbb21e1bdb9b356b1 with delete_type TABLET_DATA_DELETED (replaced by tablet c431abffd05c4136b7510c60eee2a8b1) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.793150 1165 ts_tablet_manager.cc:1918] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.793248 1165 log.cc:1199] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/8fdca99b734448118e0c42227c78e58f
I20250625 13:52:05.793520 1165 ts_tablet_manager.cc:1939] T 8fdca99b734448118e0c42227c78e58f P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.795444 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet af05fba6fe2042aab5dd47b30e4dfcf3 (table default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]). Expected version 4 got 0
W20250625 13:52:05.796104 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d8551fdf4e043cd90e2ce8228c3db7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.796799 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 8fdca99b734448118e0c42227c78e58f (table default.Table_cd4b8059082844b99439152f1be88d22 [id=d5037bf1fc844a148103da5d2ae95193]) successfully deleted
I20250625 13:52:05.797634 1170 tablet_bootstrap.cc:492] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:05.797762 1170 ts_tablet_manager.cc:1397] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.008s
I20250625 13:52:05.798586 1170 raft_consensus.cc:357] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.798807 1170 raft_consensus.cc:383] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.798888 1170 raft_consensus.cc:738] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.799131 1170 consensus_queue.cc:260] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.800262 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b7062c34fd143bf86b743055a27db8c with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd84b8afa2548fe8ce5925f0bf806f0) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.800545 884 tablet_service.cc:1515] Processing DeleteTablet for tablet a22ade8cefac4140b08882c311ea5181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.800725 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.801044 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.801450 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 637847cac1724bfeb629c4d4013396ca with delete_type TABLET_DATA_DELETED (replaced by tablet b14d00c93a964edd815122e3723f89e7) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.803158 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 575043022b1b4ca686775b2aae41a037 consensus state has no leader
I20250625 13:52:05.803412 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 43b791fd11914591bb812cc155d950dc from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.803527 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d3a17198b9a04138b24022cf8506ab36
I20250625 13:52:05.804108 596 master_service.cc:948] ReplaceTablet: received request to replace tablet af05fba6fe2042aab5dd47b30e4dfcf3 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.805398 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6b02faeef1a54d03b3fedf55f1bbbc8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.805949 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b02faeef1a54d03b3fedf55f1bbbc8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.807025 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ae4b428ae7bc49fcbc37e9d7d238ec21
I20250625 13:52:05.807649 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_50bfa9943b744384b9b3911caaf604b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.807915 1710 ts_tablet_manager.cc:1918] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.808027 1710 log.cc:1199] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/991bb428def545fd979e9f1a80ce4d18
W20250625 13:52:05.808169 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50bfa9943b744384b9b3911caaf604b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.808399 1710 ts_tablet_manager.cc:1939] T 991bb428def545fd979e9f1a80ce4d18 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.809232 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 399) took 6684 ms. Trace:
I20250625 13:52:05.809285 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 43941422259b4413bc71acfce8928e1e
I20250625 13:52:05.809340 1168 rpcz_store.cc:276] 0625 13:51:59.124811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.124867 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:05.623254 (+6498387us) ts_tablet_manager.cc:1145] Deleting tablet 34ab663b7eba452c9f9775c0f568010b
0625 13:52:05.623264 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.774789 (+151525us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.778288 (+ 3499us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.809206 (+ 30918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6476203}
I20250625 13:52:05.809892 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 400) took 6672 ms. Trace:
I20250625 13:52:05.809850 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_075afd49a8144b2eab9d29164f19a5a3" } new_table_name: "default.Table_a9488e59f8aa4ba29e11f23d2cf94b3a" modify_external_catalogs: true
I20250625 13:52:05.809966 1168 rpcz_store.cc:276] 0625 13:51:59.137674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.137725 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:05.809758 (+6672033us) ts_tablet_manager.cc:1145] Deleting tablet 3859f399a7b84de99658380bd6c24ff5
0625 13:52:05.809768 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.809875 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6661410}
I20250625 13:52:05.810444 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 401) took 6633 ms. Trace:
I20250625 13:52:05.810524 1168 rpcz_store.cc:276] 0625 13:51:59.176917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.176979 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:05.810339 (+6633360us) ts_tablet_manager.cc:1145] Deleting tablet 2de9656958164dc4ae2fb0078b38f6c3
0625 13:52:05.810349 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.810428 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6633133}
I20250625 13:52:05.810938 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 402) took 6611 ms. Trace:
I20250625 13:52:05.811019 1168 rpcz_store.cc:276] 0625 13:51:59.199676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.199742 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:05.810831 (+6611089us) ts_tablet_manager.cc:1145] Deleting tablet 69c44d72516e4883aff636ca65929c11
0625 13:52:05.810842 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.810923 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6610843}
I20250625 13:52:05.811241 1166 tablet_bootstrap.cc:492] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.811450 1166 ts_tablet_manager.cc:1397] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.011s
I20250625 13:52:05.811460 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 403) took 6501 ms. Trace:
I20250625 13:52:05.811525 1168 rpcz_store.cc:276] 0625 13:51:59.309965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.310778 (+ 813us) service_pool.cc:225] Handling call
0625 13:52:05.811356 (+6500578us) ts_tablet_manager.cc:1145] Deleting tablet d26820eaba98401da3a8b5348be21fc6
0625 13:52:05.811364 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.811445 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6497978}
I20250625 13:52:05.811551 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d373cb07af7a453dad8f774642a336dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.811944 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 407) took 6491 ms. Trace:
W20250625 13:52:05.812002 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d373cb07af7a453dad8f774642a336dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.812027 1168 rpcz_store.cc:276] 0625 13:51:59.320454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.321223 (+ 769us) service_pool.cc:225] Handling call
0625 13:52:05.811840 (+6490617us) ts_tablet_manager.cc:1145] Deleting tablet fd39216b01054ff38ec9175d7a2f87f2
0625 13:52:05.811847 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.811918 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6485645}
I20250625 13:52:05.812418 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 411) took 6476 ms. Trace:
I20250625 13:52:05.812491 1168 rpcz_store.cc:276] 0625 13:51:59.336436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.336485 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:05.812322 (+6475837us) ts_tablet_manager.cc:1145] Deleting tablet ecb9a98e402b4da8a9c2bd8317cef532
0625 13:52:05.812331 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.812403 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6475639}
I20250625 13:52:05.812513 1166 raft_consensus.cc:357] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.812669 1166 raft_consensus.cc:383] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.812716 1166 raft_consensus.cc:738] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.812870 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 415) took 6402 ms. Trace:
I20250625 13:52:05.813043 1166 consensus_queue.cc:260] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.811170 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ff091cba95574357ba76fc2e306f3666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.813907 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff091cba95574357ba76fc2e306f3666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.815294 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 34ab663b7eba452c9f9775c0f568010b (table default.Table_6a4abf185c2b44d3ba20510778eead0a [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
W20250625 13:52:05.815554 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3859f399a7b84de99658380bd6c24ff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3859f399a7b84de99658380bd6c24ff5 already in progress: creating tablet
I20250625 13:52:05.815712 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 43941422259b4413bc71acfce8928e1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.815778 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2de9656958164dc4ae2fb0078b38f6c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2de9656958164dc4ae2fb0078b38f6c3 already in progress: creating tablet
I20250625 13:52:05.812942 1168 rpcz_store.cc:276] 0625 13:51:59.410349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.410695 (+ 346us) service_pool.cc:225] Handling call
0625 13:52:05.812778 (+6402083us) ts_tablet_manager.cc:1145] Deleting tablet 430c3bed360b4f32855a8adb99076608
0625 13:52:05.812785 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.812855 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6401861}
W20250625 13:52:05.815989 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 69c44d72516e4883aff636ca65929c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69c44d72516e4883aff636ca65929c11 already in progress: creating tablet
W20250625 13:52:05.816165 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d26820eaba98401da3a8b5348be21fc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d26820eaba98401da3a8b5348be21fc6 already in progress: creating tablet
I20250625 13:52:05.816283 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 418) took 6405 ms. Trace:
W20250625 13:52:05.816380 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet fd39216b01054ff38ec9175d7a2f87f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd39216b01054ff38ec9175d7a2f87f2 already in progress: creating tablet
I20250625 13:52:05.816345 1168 rpcz_store.cc:276] 0625 13:51:59.435574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.457283 (+ 21709us) service_pool.cc:225] Handling call
0625 13:52:05.816190 (+6358907us) ts_tablet_manager.cc:1145] Deleting tablet 6f5780d88dea41e7b855b41f12e96c52
0625 13:52:05.816199 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.816269 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6353908}
W20250625 13:52:05.816576 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ecb9a98e402b4da8a9c2bd8317cef532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb9a98e402b4da8a9c2bd8317cef532 already in progress: creating tablet
I20250625 13:52:05.816689 884 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.816769 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 430c3bed360b4f32855a8adb99076608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 430c3bed360b4f32855a8adb99076608 already in progress: creating tablet
I20250625 13:52:05.816854 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a17198b9a04138b24022cf8506ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:05.816965 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6f5780d88dea41e7b855b41f12e96c52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f5780d88dea41e7b855b41f12e96c52 already in progress: creating tablet
I20250625 13:52:05.817189 1166 ts_tablet_manager.cc:1428] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:05.817448 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a17198b9a04138b24022cf8506ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.817327 1168 tablet_replica.cc:331] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:05.817981 1168 raft_consensus.cc:2241] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:05.818017 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.818174 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 43941422259b4413bc71acfce8928e1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.818317 1168 raft_consensus.cc:2270] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.820768 1166 tablet_bootstrap.cc:492] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.822383 1166 tablet_bootstrap.cc:654] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.822695 1171 tablet_bootstrap.cc:492] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.825181 1171 tablet_bootstrap.cc:654] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.831837 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 9a31f10c8ca745b7ad635c622a265450 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:05.832356 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2ed47131082c4d11b7ba70c6432bc36c" } new_table_name: "default.Table_0217bc55d72a4cfd8f9e87b2ae77d106" modify_external_catalogs: true
I20250625 13:52:05.825728 743 tablet_service.cc:1515] Processing DeleteTablet for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.826279 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 43941422259b4413bc71acfce8928e1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.829748 757 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a17198b9a04138b24022cf8506ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.833745 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 43941422259b4413bc71acfce8928e1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43941422259b4413bc71acfce8928e1e already in progress: creating tablet
I20250625 13:52:05.833887 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e77736191544e8da32ca9927f0a5804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.833978 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d3a17198b9a04138b24022cf8506ab36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a17198b9a04138b24022cf8506ab36 already in progress: creating tablet
W20250625 13:52:05.834142 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae4b428ae7bc49fcbc37e9d7d238ec21 already in progress: creating tablet
I20250625 13:52:05.834161 1165 tablet_replica.cc:331] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.831574 1168 ts_tablet_manager.cc:1905] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.834319 1165 raft_consensus.cc:2241] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.834467 1165 raft_consensus.cc:2270] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.835188 1165 ts_tablet_manager.cc:1905] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.837095 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b4be0bc23cde4f0f94bd678096e0c7d8
I20250625 13:52:05.840041 743 tablet_service.cc:1515] Processing DeleteTablet for tablet b4be0bc23cde4f0f94bd678096e0c7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.840725 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet b4be0bc23cde4f0f94bd678096e0c7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.840731 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b4be0bc23cde4f0f94bd678096e0c7d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.843768 1165 ts_tablet_manager.cc:1918] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.843924 1165 log.cc:1199] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/0e77736191544e8da32ca9927f0a5804
I20250625 13:52:05.844242 1165 ts_tablet_manager.cc:1939] T 0e77736191544e8da32ca9927f0a5804 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:05.844380 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_2dbd2e7b0a624bcb99f8edc7918b558d [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 2 got 0
I20250625 13:52:05.844830 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
I20250625 13:52:05.845387 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 0e77736191544e8da32ca9927f0a5804 (table default.Table_b9f454bddd8a463b863eb0ca65ebf3af [id=7467e5b0513d42ff8c886b6080a5b539]) successfully deleted
I20250625 13:52:05.847707 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 29be8c0350584de382985e66556bdcca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.851483 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e77736191544e8da32ca9927f0a5804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.857600 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet b4be0bc23cde4f0f94bd678096e0c7d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4be0bc23cde4f0f94bd678096e0c7d8 already in progress: creating tablet
W20250625 13:52:05.859045 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0e77736191544e8da32ca9927f0a5804 because the tablet was not found. No further retry: Not found: Tablet not found: 0e77736191544e8da32ca9927f0a5804
I20250625 13:52:05.859150 1170 ts_tablet_manager.cc:1428] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20250625 13:52:05.859431 1821 raft_consensus.cc:491] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.859524 1166 tablet_bootstrap.cc:492] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:05.859555 1821 raft_consensus.cc:513] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.859660 1166 ts_tablet_manager.cc:1397] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250625 13:52:05.860145 1821 leader_election.cc:290] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.860381 1166 raft_consensus.cc:357] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.861534 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2dbd2e7b0a624bcb99f8edc7918b558d" } new_table_name: "default.Table_d6bab22e154a444f90a895591fa09ca3" modify_external_catalogs: true
W20250625 13:52:05.861824 985 leader_election.cc:343] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 27092991fcad475fabc77d1f8bab0582
I20250625 13:52:05.863338 1166 raft_consensus.cc:383] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:52:05.863384 983 leader_election.cc:343] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 27092991fcad475fabc77d1f8bab0582
I20250625 13:52:05.863483 983 leader_election.cc:304] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:05.863492 1166 raft_consensus.cc:738] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.863718 1166 consensus_queue.cc:260] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.864205 1166 ts_tablet_manager.cc:1428] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250625 13:52:05.864459 1166 tablet_bootstrap.cc:492] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:05.866725 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f4fee43fefb74cd5bd6e7cc25308e969"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:05.867842 1821 raft_consensus.cc:2747] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:05.867247 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4fee43fefb74cd5bd6e7cc25308e969 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.869769 1710 tablet_replica.cc:331] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.869944 1710 raft_consensus.cc:2241] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.870082 1710 raft_consensus.cc:2270] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.870569 1710 ts_tablet_manager.cc:1905] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.870972 1166 tablet_bootstrap.cc:654] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.873790 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 43b791fd11914591bb812cc155d950dc
I20250625 13:52:05.873980 1171 tablet_bootstrap.cc:492] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:05.874110 1171 ts_tablet_manager.cc:1397] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20250625 13:52:05.874097 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 991bb428def545fd979e9f1a80ce4d18 (table default.Table_a21873ae3bbd4a1aaf287e80fe14c12e [id=fd9a1350f16e4c54a613cf8f7026240a]) successfully deleted
I20250625 13:52:05.874315 593 catalog_manager.cc:6652] ReplaceTablet: tablet 43b791fd11914591bb812cc155d950dc deleted and replaced by tablet c0417fae218a4ec8bfd18f4834a0789f
W20250625 13:52:05.874331 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet da2f8d0d9a714a28ba6fb9642d461de2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da2f8d0d9a714a28ba6fb9642d461de2 already in progress: creating tablet
W20250625 13:52:05.875098 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 58a532e889244b088562081e444234a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58a532e889244b088562081e444234a3 already in progress: creating tablet
I20250625 13:52:05.875763 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b791fd11914591bb812cc155d950dc with delete_type TABLET_DATA_DELETED (replaced by tablet c0417fae218a4ec8bfd18f4834a0789f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.876168 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b791fd11914591bb812cc155d950dc with delete_type TABLET_DATA_DELETED (replaced by tablet c0417fae218a4ec8bfd18f4834a0789f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.877564 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fc57b2c273df4835bd06effe454ccc27" } new_table_name: "default.Table_9c9bfac37ad24311921d77442fc74013" modify_external_catalogs: true
I20250625 13:52:05.884898 1170 tablet_bootstrap.cc:492] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:05.886087 1748 consensus_queue.cc:1035] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:05.886374 1710 ts_tablet_manager.cc:1918] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:05.886478 1710 log.cc:1199] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/19017cc372af43b2b5074a8990375ede
I20250625 13:52:05.886785 1710 ts_tablet_manager.cc:1939] T 19017cc372af43b2b5074a8990375ede P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:05.887320 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet af05fba6fe2042aab5dd47b30e4dfcf3
I20250625 13:52:05.887790 596 catalog_manager.cc:6652] ReplaceTablet: tablet af05fba6fe2042aab5dd47b30e4dfcf3 deleted and replaced by tablet abc44a41b30e4a53977ec81d6905612f
I20250625 13:52:05.888332 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet af05fba6fe2042aab5dd47b30e4dfcf3 with delete_type TABLET_DATA_DELETED (replaced by tablet abc44a41b30e4a53977ec81d6905612f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.888833 893 tablet_service.cc:1515] Processing DeleteTablet for tablet af05fba6fe2042aab5dd47b30e4dfcf3 with delete_type TABLET_DATA_DELETED (replaced by tablet abc44a41b30e4a53977ec81d6905612f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.889312 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6d7be80f24cc434c949137a402b4b7c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.889874 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d7be80f24cc434c949137a402b4b7c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:05.892882 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a780f2d2daf465a940d364477139b73 consensus state has no leader
I20250625 13:52:05.893059 1776 raft_consensus.cc:491] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.893227 1776 raft_consensus.cc:513] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.893841 1170 tablet_bootstrap.cc:654] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.893934 1776 leader_election.cc:290] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.894673 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e529f5bf13f4ee69e66402b1ee252d4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:05.895064 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e529f5bf13f4ee69e66402b1ee252d4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:05.895975 711 leader_election.cc:343] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:05.896157 711 leader_election.cc:343] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.896227 711 leader_election.cc:304] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:05.896723 1776 raft_consensus.cc:2747] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.897883 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 19017cc372af43b2b5074a8990375ede (table default.Table_d675ca14ac884a35aab70e66cdcef846 [id=fed9fb5422bc435ab14ed8f178cc95a1]) successfully deleted
I20250625 13:52:05.898104 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b791fd11914591bb812cc155d950dc with delete_type TABLET_DATA_DELETED (replaced by tablet c0417fae218a4ec8bfd18f4834a0789f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.898311 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 1effed0fa10d409abcb31dbba54861d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:05.898481 746 tablet_service.cc:1515] Processing DeleteTablet for tablet af05fba6fe2042aab5dd47b30e4dfcf3 with delete_type TABLET_DATA_DELETED (replaced by tablet abc44a41b30e4a53977ec81d6905612f) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:05.899258 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c4db530b54f04fedb18bf993dc60d8da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4db530b54f04fedb18bf993dc60d8da already in progress: creating tablet
W20250625 13:52:05.899926 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 00d4acd387c84d55834f632f41045862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00d4acd387c84d55834f632f41045862 already in progress: creating tablet
I20250625 13:52:05.900233 1710 tablet_replica.cc:331] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.900449 1710 raft_consensus.cc:2241] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.900707 1710 raft_consensus.cc:2270] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.874818 1171 raft_consensus.cc:357] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.901297 1171 raft_consensus.cc:383] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.901397 1171 raft_consensus.cc:738] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.901978 1171 consensus_queue.cc:260] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.904825 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_2dbd2e7b0a624bcb99f8edc7918b558d [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 2 got 0
I20250625 13:52:05.905381 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e47fa6053bb148778065f7d411fb935a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:05.906082 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e47fa6053bb148778065f7d411fb935a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:05.907610 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c0a3ea18bab944749d45da9b3139f8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:05.909567 1168 ts_tablet_manager.cc:1918] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:05.909680 1168 log.cc:1199] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/cf1664afbab647b3baeed6391196bed9
I20250625 13:52:05.910063 1168 ts_tablet_manager.cc:1939] T cf1664afbab647b3baeed6391196bed9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:05.910828 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 420) took 6311 ms. Trace:
I20250625 13:52:05.910918 1168 rpcz_store.cc:276] 0625 13:51:59.598918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.598993 (+ 75us) service_pool.cc:225] Handling call
0625 13:52:05.817279 (+6218286us) ts_tablet_manager.cc:1145] Deleting tablet cf1664afbab647b3baeed6391196bed9
0625 13:52:05.817289 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:05.848535 (+ 31246us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.909448 (+ 60913us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:05.910812 (+ 1364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6217453}
I20250625 13:52:05.911355 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet cf1664afbab647b3baeed6391196bed9 (table default.Table_98ae9c7310a24e8d988b919a5ec496e7 [id=05b9dac188b34d0b8022defddccbff26]) successfully deleted
W20250625 13:52:05.912264 849 consensus_peers.cc:487] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:05.916426 1165 tablet_replica.cc:331] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:05.916554 1170 tablet_bootstrap.cc:492] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:05.916754 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 43b791fd11914591bb812cc155d950dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b791fd11914591bb812cc155d950dc already in progress: creating tablet
I20250625 13:52:05.916808 1170 ts_tablet_manager.cc:1397] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.003s
I20250625 13:52:05.916940 1165 raft_consensus.cc:2241] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.917184 1165 raft_consensus.cc:2270] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.917551 1170 raft_consensus.cc:357] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.917724 1170 raft_consensus.cc:383] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:05.917773 1170 raft_consensus.cc:738] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:05.917992 1170 consensus_queue.cc:260] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.920049 1165 ts_tablet_manager.cc:1905] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.920082 1837 raft_consensus.cc:491] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.920204 1837 raft_consensus.cc:513] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.920847 1837 leader_election.cc:290] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.921849 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:05.921958 1170 ts_tablet_manager.cc:1428] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
W20250625 13:52:05.922739 849 leader_election.cc:343] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.923550 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_2dbd2e7b0a624bcb99f8edc7918b558d [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 2 got 0
I20250625 13:52:05.923585 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:05.923825 777 raft_consensus.cc:2466] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:05.924209 851 leader_election.cc:304] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:05.924624 1842 raft_consensus.cc:2802] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:05.924695 1842 raft_consensus.cc:491] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:05.924748 1842 raft_consensus.cc:3058] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.925580 1710 ts_tablet_manager.cc:1905] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.927696 1171 ts_tablet_manager.cc:1428] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.054s user 0.002s sys 0.000s
I20250625 13:52:05.927994 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 6771d8e180fe48569a0e016db0cc9b51 with delete_type TABLET_DATA_DELETED (replaced by tablet 86af0ea6648045a1ac4ba903d20849b7) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:05.928119 1170 tablet_bootstrap.cc:492] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:05.929911 1170 tablet_bootstrap.cc:654] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:05.930032 1842 raft_consensus.cc:513] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:05.936244 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0012e95ff4924723a6e6aaf6b85da438 consensus state has no leader
I20250625 13:52:05.936395 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:05.936692 777 raft_consensus.cc:3058] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.939549 777 raft_consensus.cc:2466] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:05.939985 851 leader_election.cc:304] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:05.940279 1748 raft_consensus.cc:2802] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:05.940366 1748 raft_consensus.cc:695] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:05.940703 1748 consensus_queue.cc:237] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.935125 1842 leader_election.cc:290] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:05.936103 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:05.943919 849 leader_election.cc:343] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:05.945886 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0965fa983950458882df05ba9d414275" } modify_external_catalogs: true
I20250625 13:52:05.946086 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0965fa983950458882df05ba9d414275" } modify_external_catalogs: true
I20250625 13:52:05.947122 1837 raft_consensus.cc:491] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.947264 1837 raft_consensus.cc:513] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.947845 1837 leader_election.cc:290] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:05.948119 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11711546f0e14f9a9eabf003f49a69b7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.948622 849 leader_election.cc:343] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:05.948799 851 leader_election.cc:343] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 11711546f0e14f9a9eabf003f49a69b7
I20250625 13:52:05.948940 851 leader_election.cc:304] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.949115 1837 raft_consensus.cc:491] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.949151 1510 raft_consensus.cc:2747] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.949203 1837 raft_consensus.cc:513] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:05.949777 1837 leader_election.cc:290] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:05.950035 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "58aa3c21191e4bea9c5395d6e32b8717" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:05.950501 849 leader_election.cc:343] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:05.950762 851 leader_election.cc:343] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 58aa3c21191e4bea9c5395d6e32b8717
I20250625 13:52:05.950981 851 leader_election.cc:304] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.951710 1837 raft_consensus.cc:2747] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:05.956808 1821 raft_consensus.cc:491] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:05.956986 1821 raft_consensus.cc:513] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.957715 1821 leader_election.cc:290] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.958806 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01ba67b500b646d7aa55315f7ee23ed6" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:05.958936 985 leader_election.cc:343] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 01ba67b500b646d7aa55315f7ee23ed6
I20250625 13:52:05.958976 912 raft_consensus.cc:2466] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:05.959378 983 leader_election.cc:304] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.959892 1821 raft_consensus.cc:2802] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:05.959957 1821 raft_consensus.cc:491] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:05.960008 1821 raft_consensus.cc:3058] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.963152 1821 raft_consensus.cc:513] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:05.964627 985 leader_election.cc:343] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 01ba67b500b646d7aa55315f7ee23ed6
I20250625 13:52:05.964807 1821 leader_election.cc:290] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:05.965305 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01ba67b500b646d7aa55315f7ee23ed6" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:05.965543 912 raft_consensus.cc:3058] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:05.934942 1171 tablet_bootstrap.cc:492] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:05.968628 912 raft_consensus.cc:2466] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:05.969003 983 leader_election.cc:304] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:05.969570 1821 raft_consensus.cc:2802] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:05.973724 1821 raft_consensus.cc:695] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:05.974722 1710 ts_tablet_manager.cc:1918] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:05.974833 1710 log.cc:1199] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/b1acc41a93f6433588727811903921fe
I20250625 13:52:05.975024 1821 consensus_queue.cc:237] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:05.976080 1710 ts_tablet_manager.cc:1939] T b1acc41a93f6433588727811903921fe P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:05.977330 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 consensus state has no leader
I20250625 13:52:05.980782 1929 tablet.cc:1722] T d8b30c229c3445b683a5015f069eca1c P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:05.983722 1710 tablet_replica.cc:331] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:05.984349 1710 raft_consensus.cc:2241] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:05.985144 1710 raft_consensus.cc:2270] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:05.985674 1710 ts_tablet_manager.cc:1905] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:05.985669 593 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_d6bab22e154a444f90a895591fa09ca3 [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 3 got 0
I20250625 13:52:05.987604 1931 tablet.cc:1722] T d8b30c229c3445b683a5015f069eca1c P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:05.989236 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]) successfully deleted
W20250625 13:52:05.989570 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f065b4e3218b4e6e8a2ac3a70eede13b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f065b4e3218b4e6e8a2ac3a70eede13b already in progress: creating tablet
I20250625 13:52:05.993618 1046 raft_consensus.cc:1273] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Refusing update from remote peer affa1c3035b940e88ca143be7d35840d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:05.993953 777 raft_consensus.cc:1273] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer affa1c3035b940e88ca143be7d35840d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:05.994320 1510 consensus_queue.cc:1035] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:05.996579 1510 consensus_queue.cc:1035] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:06.008642 1171 tablet_bootstrap.cc:654] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.015136 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 9252dde349d24a8c99dfe2eda37c9f64 (DEFAULT_TABLE table=default.Table_feec880c1f294ed2a18e041a5bc7a405 [id=4786c25d2f5c4cf4b34abe26d28b90ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.015642 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9252dde349d24a8c99dfe2eda37c9f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.015578 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_65b9f8c4792145df86eba8b5f47bf1a5" } modify_external_catalogs: true
I20250625 13:52:06.015801 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_65b9f8c4792145df86eba8b5f47bf1a5" } modify_external_catalogs: true
I20250625 13:52:06.018042 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 421) took 6403 ms. Trace:
I20250625 13:52:06.018131 1168 rpcz_store.cc:276] 0625 13:51:59.614489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.614547 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:06.017904 (+6403357us) ts_tablet_manager.cc:1145] Deleting tablet 413afd7b62d24fe8b9eceadd57e29fc8
0625 13:52:06.017913 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.018021 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6403118}
I20250625 13:52:06.018590 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5f49c0fc97d44b28acfd6cb493906edb
W20250625 13:52:06.018627 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 413afd7b62d24fe8b9eceadd57e29fc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 413afd7b62d24fe8b9eceadd57e29fc8 already in progress: creating tablet
I20250625 13:52:06.019151 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 422) took 6403 ms. Trace:
I20250625 13:52:06.019224 1168 rpcz_store.cc:276] 0625 13:51:59.615259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.615333 (+ 74us) service_pool.cc:225] Handling call
0625 13:52:06.019042 (+6403709us) ts_tablet_manager.cc:1145] Deleting tablet abd6f9121a9d4b39adde1b880a875e23
0625 13:52:06.019053 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.019134 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6403516}
I20250625 13:52:06.019580 1168 tablet_replica.cc:331] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:06.019712 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f49c0fc97d44b28acfd6cb493906edb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.019760 1168 raft_consensus.cc:2241] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:06.020016 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f49c0fc97d44b28acfd6cb493906edb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:06.020540 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet abd6f9121a9d4b39adde1b880a875e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abd6f9121a9d4b39adde1b880a875e23 already in progress: creating tablet
I20250625 13:52:06.021203 1166 tablet_bootstrap.cc:492] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.021324 1166 ts_tablet_manager.cc:1397] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.157s user 0.009s sys 0.014s
I20250625 13:52:06.021250 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f471ebec4d67405899f155aad9d3ec83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.021745 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f471ebec4d67405899f155aad9d3ec83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.022162 1166 raft_consensus.cc:357] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.022341 1166 raft_consensus.cc:383] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.022389 1166 raft_consensus.cc:738] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.022568 1166 consensus_queue.cc:260] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.023070 1166 ts_tablet_manager.cc:1428] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:06.023020 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ee7ffdafe40d48bc89ffa9a18595cf9d" } modify_external_catalogs: true
I20250625 13:52:06.023180 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ee7ffdafe40d48bc89ffa9a18595cf9d" } modify_external_catalogs: true
I20250625 13:52:06.023334 1166 tablet_bootstrap.cc:492] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.023607 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f49c0fc97d44b28acfd6cb493906edb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.026877 746 tablet_service.cc:1468] Processing CreateTablet for tablet 9252dde349d24a8c99dfe2eda37c9f64 (DEFAULT_TABLE table=default.Table_feec880c1f294ed2a18e041a5bc7a405 [id=4786c25d2f5c4cf4b34abe26d28b90ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.027251 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9252dde349d24a8c99dfe2eda37c9f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.027237 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9252dde349d24a8c99dfe2eda37c9f64 (DEFAULT_TABLE table=default.Table_feec880c1f294ed2a18e041a5bc7a405 [id=4786c25d2f5c4cf4b34abe26d28b90ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.027611 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9252dde349d24a8c99dfe2eda37c9f64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.027763 1165 ts_tablet_manager.cc:1918] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.027850 1165 log.cc:1199] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/1effed0fa10d409abcb31dbba54861d6
I20250625 13:52:06.028113 1165 ts_tablet_manager.cc:1939] T 1effed0fa10d409abcb31dbba54861d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:06.029014 1165 tablet_replica.cc:331] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:06.029171 1165 raft_consensus.cc:2241] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.029291 1165 raft_consensus.cc:2270] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.030689 1171 tablet_bootstrap.cc:492] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:06.030810 1171 ts_tablet_manager.cc:1397] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.002s
I20250625 13:52:06.031697 1171 raft_consensus.cc:357] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.031904 1171 raft_consensus.cc:383] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.032014 1171 raft_consensus.cc:738] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.032091 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 1effed0fa10d409abcb31dbba54861d6 (table default.Table_ec42501f607c41ceb9d35f3dd2a38f34 [id=d2c6242f48434d3bb6c2dc01fa095a67]) successfully deleted
I20250625 13:52:06.032321 1171 consensus_queue.cc:260] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.020059 1168 pending_rounds.cc:70] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:06.033597 1168 pending_rounds.cc:77] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520595703603200 op_type: NO_OP noop_request { }
I20250625 13:52:06.033787 1168 raft_consensus.cc:2887] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:06.033937 1168 raft_consensus.cc:2270] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.034860 1170 tablet_bootstrap.cc:492] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.035077 1170 ts_tablet_manager.cc:1397] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.107s user 0.014s sys 0.020s
I20250625 13:52:06.035634 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 5788c3143f8c4368a0a765a1827134a9 (DEFAULT_TABLE table=default.Table_40856c021b18443fb5feae09a82e09ad [id=49b35ec368d24365a560c14e2ba753cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.036062 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5788c3143f8c4368a0a765a1827134a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.036659 1170 raft_consensus.cc:357] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.037016 1170 raft_consensus.cc:383] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.037112 1170 raft_consensus.cc:738] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.037495 1170 consensus_queue.cc:260] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.037987 1170 ts_tablet_manager.cc:1428] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:06.038347 1170 tablet_bootstrap.cc:492] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.040481 1170 tablet_bootstrap.cc:654] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.042495 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a637d32e43d647768eebfd52dd249a2c" } modify_external_catalogs: true
I20250625 13:52:06.042728 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a637d32e43d647768eebfd52dd249a2c" } modify_external_catalogs: true
I20250625 13:52:06.045190 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5788c3143f8c4368a0a765a1827134a9 (DEFAULT_TABLE table=default.Table_40856c021b18443fb5feae09a82e09ad [id=49b35ec368d24365a560c14e2ba753cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.045691 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5788c3143f8c4368a0a765a1827134a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.046059 746 tablet_service.cc:1468] Processing CreateTablet for tablet 5788c3143f8c4368a0a765a1827134a9 (DEFAULT_TABLE table=default.Table_40856c021b18443fb5feae09a82e09ad [id=49b35ec368d24365a560c14e2ba753cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.047624 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f04ee0e6b3324e169e99b807c1faa3a0" } modify_external_catalogs: true
I20250625 13:52:06.047871 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f04ee0e6b3324e169e99b807c1faa3a0" } modify_external_catalogs: true
I20250625 13:52:06.049072 594 catalog_manager.cc:5582] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:06.050413 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5788c3143f8c4368a0a765a1827134a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.054391 1710 ts_tablet_manager.cc:1918] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.049759 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0217bc55d72a4cfd8f9e87b2ae77d106" } new_table_name: "default.Table_a382ff2909274bbd88d364ade2fe1080" modify_external_catalogs: true
I20250625 13:52:06.054957 1710 log.cc:1199] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/27092991fcad475fabc77d1f8bab0582
I20250625 13:52:06.055269 1710 ts_tablet_manager.cc:1939] T 27092991fcad475fabc77d1f8bab0582 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.056639 1023 tablet_service.cc:1468] Processing CreateTablet for tablet e5592ab667b04932896addf7514af24e (DEFAULT_TABLE table=default.Table_d8906d18878c49f59fe57ee84374beb9 [id=48a8063c68914a71bcc4fcdb97d7a0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.058192 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d7c4a6a4fd574012818f3aa0a8b4d71f
I20250625 13:52:06.058815 757 tablet_service.cc:1515] Processing DeleteTablet for tablet d7c4a6a4fd574012818f3aa0a8b4d71f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.050213 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_64d842d3737b421f9832c07a67be66e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.054827 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
W20250625 13:52:06.059410 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64d842d3737b421f9832c07a67be66e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.063941 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 17efe97d559a45dc9c0ca0b74efa7aa6
I20250625 13:52:06.065029 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.066015 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
I20250625 13:52:06.066335 1168 ts_tablet_manager.cc:1905] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.067663 1171 ts_tablet_manager.cc:1428] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.037s user 0.000s sys 0.002s
I20250625 13:52:06.067934 1171 tablet_bootstrap.cc:492] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:06.068960 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_42a2e34e5d204e7188555f5e5616cc4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.069624 1929 tablet.cc:1722] T c7c3f0ca6fa5482db7f57143911c838f P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
W20250625 13:52:06.069475 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42a2e34e5d204e7188555f5e5616cc4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.070132 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet d7c4a6a4fd574012818f3aa0a8b4d71f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.070878 884 tablet_service.cc:1515] Processing DeleteTablet for tablet d7c4a6a4fd574012818f3aa0a8b4d71f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.071497 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.071848 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:06.072319 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.073175 757 tablet_service.cc:1468] Processing CreateTablet for tablet e5592ab667b04932896addf7514af24e (DEFAULT_TABLE table=default.Table_d8906d18878c49f59fe57ee84374beb9 [id=48a8063c68914a71bcc4fcdb97d7a0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.073510 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5592ab667b04932896addf7514af24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.073567 1931 tablet.cc:1722] T c7c3f0ca6fa5482db7f57143911c838f P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250625 13:52:06.084880 1170 tablet_bootstrap.cc:492] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.085196 1170 ts_tablet_manager.cc:1397] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.009s
I20250625 13:52:06.086506 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 614181c6e4d24e7b9e5da3b4818cf065
I20250625 13:52:06.088393 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 614181c6e4d24e7b9e5da3b4818cf065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.088843 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 614181c6e4d24e7b9e5da3b4818cf065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.089509 1170 raft_consensus.cc:357] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.089769 1170 raft_consensus.cc:383] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.089879 1170 raft_consensus.cc:738] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.090387 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 614181c6e4d24e7b9e5da3b4818cf065 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.090508 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 27092991fcad475fabc77d1f8bab0582 (table default.Table_51ca9ca8018f4b2cbcf3c97ae4e2f7ba [id=540530d4619a4093b20d1bcba9ba99ac]) successfully deleted
I20250625 13:52:06.090174 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5592ab667b04932896addf7514af24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.094178 757 tablet_service.cc:1468] Processing CreateTablet for tablet d9b9c0a07c084d9dbba745f6522e3e57 (DEFAULT_TABLE table=default.Table_29c1207ed89e4a33bceb77bc37580565 [id=0ea61479d0cc4670b609dd1d588b7cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.094551 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b9c0a07c084d9dbba745f6522e3e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.096092 884 tablet_service.cc:1468] Processing CreateTablet for tablet e5592ab667b04932896addf7514af24e (DEFAULT_TABLE table=default.Table_d8906d18878c49f59fe57ee84374beb9 [id=48a8063c68914a71bcc4fcdb97d7a0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.096429 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5592ab667b04932896addf7514af24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.103545 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_034f6cf597cd469fba12687016eff7d1" } new_table_name: "default.Table_f618b45e019044ea946538ed3fa05471" modify_external_catalogs: true
I20250625 13:52:06.104240 881 tablet_service.cc:1468] Processing CreateTablet for tablet d9b9c0a07c084d9dbba745f6522e3e57 (DEFAULT_TABLE table=default.Table_29c1207ed89e4a33bceb77bc37580565 [id=0ea61479d0cc4670b609dd1d588b7cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.104602 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b9c0a07c084d9dbba745f6522e3e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.105259 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0b1631a468c34b49a62449290f63ea32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.091130 1710 tablet_replica.cc:331] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:06.105928 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b1631a468c34b49a62449290f63ea32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.106108 1710 raft_consensus.cc:2241] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:06.106333 1710 pending_rounds.cc:70] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:06.106374 1710 pending_rounds.cc:77] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520618396721152 op_type: NO_OP noop_request { }
I20250625 13:52:06.106451 1710 raft_consensus.cc:2887] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:06.106516 1710 raft_consensus.cc:2270] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.106817 1170 consensus_queue.cc:260] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.107257 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5c1dbe2671ff41d9b8d115b4b7ea58d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.107429 1170 ts_tablet_manager.cc:1428] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250625 13:52:06.107724 1170 tablet_bootstrap.cc:492] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:06.107722 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c1dbe2671ff41d9b8d115b4b7ea58d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.109158 1710 ts_tablet_manager.cc:1905] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.110930 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_865d72bbadc14727a2bd54342b385ee3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.111379 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_865d72bbadc14727a2bd54342b385ee3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.112268 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d9b9c0a07c084d9dbba745f6522e3e57 (DEFAULT_TABLE table=default.Table_29c1207ed89e4a33bceb77bc37580565 [id=0ea61479d0cc4670b609dd1d588b7cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.113549 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d7704b38144429b9c95675be84a4730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.115136 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6e147e09b5244a8b8f52e8d10c3db4c1
I20250625 13:52:06.116055 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b9c0a07c084d9dbba745f6522e3e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.116259 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.118561 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.118669 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.119992 1170 tablet_bootstrap.cc:654] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.121179 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1d9bd351536c4558b189cebf8b793343 (DEFAULT_TABLE table=default.Table_5821b12a3aff40dcb22e592583f8c1f5 [id=0d58edf3a60147bebc2620f6e5af56fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.121796 746 tablet_service.cc:1468] Processing CreateTablet for tablet 1d9bd351536c4558b189cebf8b793343 (DEFAULT_TABLE table=default.Table_5821b12a3aff40dcb22e592583f8c1f5 [id=0d58edf3a60147bebc2620f6e5af56fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.122217 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9bd351536c4558b189cebf8b793343. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:06.124922 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0012e95ff4924723a6e6aaf6b85da438 consensus state has no leader
I20250625 13:52:06.127321 893 tablet_service.cc:1468] Processing CreateTablet for tablet 1d9bd351536c4558b189cebf8b793343 (DEFAULT_TABLE table=default.Table_5821b12a3aff40dcb22e592583f8c1f5 [id=0d58edf3a60147bebc2620f6e5af56fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.127841 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9bd351536c4558b189cebf8b793343. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.128332 1166 tablet_bootstrap.cc:654] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.130715 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9bd351536c4558b189cebf8b793343. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.132683 1837 raft_consensus.cc:491] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.132812 1837 raft_consensus.cc:513] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.133101 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a2acecefed724eb3b55116b8e5452853 (DEFAULT_TABLE table=default.Table_e23d05f0e5074305a2cdbd567759c6d3 [id=40996d95a7cb402194bb2c6372e21d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.133373 1837 leader_election.cc:290] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.133447 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2acecefed724eb3b55116b8e5452853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.134232 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29be8c0350584de382985e66556bdcca" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.134799 849 leader_election.cc:343] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:06.135596 851 leader_election.cc:343] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 29be8c0350584de382985e66556bdcca
I20250625 13:52:06.135701 851 leader_election.cc:304] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.136125 1837 raft_consensus.cc:2747] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.140061 1171 tablet_bootstrap.cc:654] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.141881 746 tablet_service.cc:1468] Processing CreateTablet for tablet a2acecefed724eb3b55116b8e5452853 (DEFAULT_TABLE table=default.Table_e23d05f0e5074305a2cdbd567759c6d3 [id=40996d95a7cb402194bb2c6372e21d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.141963 881 tablet_service.cc:1468] Processing CreateTablet for tablet a2acecefed724eb3b55116b8e5452853 (DEFAULT_TABLE table=default.Table_e23d05f0e5074305a2cdbd567759c6d3 [id=40996d95a7cb402194bb2c6372e21d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.142378 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2acecefed724eb3b55116b8e5452853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.143961 1165 ts_tablet_manager.cc:1905] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.144392 1837 raft_consensus.cc:491] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.144546 1837 raft_consensus.cc:513] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.144937 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2acecefed724eb3b55116b8e5452853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.145165 1837 leader_election.cc:290] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.145907 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbd2120b4b084691b6604305773e74f9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.146799 849 leader_election.cc:343] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:06.147467 851 leader_election.cc:343] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: dbd2120b4b084691b6604305773e74f9
I20250625 13:52:06.147653 1942 tablet.cc:1722] T c7c3f0ca6fa5482db7f57143911c838f P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250625 13:52:06.147799 851 leader_election.cc:304] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.148660 1837 raft_consensus.cc:2747] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.154052 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e47fa6053bb148778065f7d411fb935a" } new_table_name: "default.Table_e5270cb847db4a57979fd35c13f6fe0b" modify_external_catalogs: true
I20250625 13:52:06.156101 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0a8b5ac0e4a447a6b313a397677475fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.156886 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a8b5ac0e4a447a6b313a397677475fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.165452 1821 raft_consensus.cc:491] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.165634 1821 raft_consensus.cc:513] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.166345 1821 leader_election.cc:290] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:06.167577 746 tablet_service.cc:1468] Processing CreateTablet for tablet 529c3bf8e72f46948cfb864445ff51f3 (DEFAULT_TABLE table=default.Table_d0ce3bb2d8b04255ae3bc517960db379 [id=5227457618334dd0ad007356f6a79539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.168133 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529c3bf8e72f46948cfb864445ff51f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.168715 1166 tablet_bootstrap.cc:492] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.168860 1166 ts_tablet_manager.cc:1397] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.146s user 0.004s sys 0.000s
I20250625 13:52:06.169564 1166 raft_consensus.cc:357] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.169737 1166 raft_consensus.cc:383] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.169796 1166 raft_consensus.cc:738] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.169989 1166 consensus_queue.cc:260] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.170589 1166 ts_tablet_manager.cc:1428] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:06.172086 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae4a105e80c1460cbb21e1bdb9b356b1" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:06.172178 743 tablet_service.cc:1468] Processing CreateTablet for tablet 71170aaf26ed4f1b8c458d90db6e2765 (DEFAULT_TABLE table=default.Table_931f25deb4904d2fa74ab9437e733369 [id=36b6dd7a254247679b0eb5420e25a376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.172294 912 raft_consensus.cc:2466] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:06.172765 983 leader_election.cc:304] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:06.173035 1821 raft_consensus.cc:2802] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:06.173112 1821 raft_consensus.cc:491] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:06.173175 1821 raft_consensus.cc:3058] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:06.173363 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_d6bab22e154a444f90a895591fa09ca3 [id=5ad3bfe6145f4a4cbc7dc4a31cc43507] alter complete (version 3)
I20250625 13:52:06.174095 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 36b151e200e1430c99ac194bc55881c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.175971 892 tablet_service.cc:1468] Processing CreateTablet for tablet 71170aaf26ed4f1b8c458d90db6e2765 (DEFAULT_TABLE table=default.Table_931f25deb4904d2fa74ab9437e733369 [id=36b6dd7a254247679b0eb5420e25a376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.176321 1821 raft_consensus.cc:513] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.177663 1046 raft_consensus.cc:3058] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:06.182413 1710 ts_tablet_manager.cc:1918] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:06.182533 1710 log.cc:1199] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/01ba67b500b646d7aa55315f7ee23ed6
I20250625 13:52:06.183004 1821 leader_election.cc:290] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:06.183161 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae4a105e80c1460cbb21e1bdb9b356b1" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:06.183367 912 raft_consensus.cc:3058] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:06.175697 884 tablet_service.cc:1468] Processing CreateTablet for tablet 529c3bf8e72f46948cfb864445ff51f3 (DEFAULT_TABLE table=default.Table_d0ce3bb2d8b04255ae3bc517960db379 [id=5227457618334dd0ad007356f6a79539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.183674 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529c3bf8e72f46948cfb864445ff51f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.186324 912 raft_consensus.cc:2466] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:06.186682 1166 tablet_bootstrap.cc:492] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.188205 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71170aaf26ed4f1b8c458d90db6e2765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.188498 1166 tablet_bootstrap.cc:654] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.189133 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71170aaf26ed4f1b8c458d90db6e2765. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:06.192694 985 leader_election.cc:343] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ae4a105e80c1460cbb21e1bdb9b356b1
I20250625 13:52:06.192795 983 leader_election.cc:304] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:06.193002 985 leader_election.cc:343] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ae4a105e80c1460cbb21e1bdb9b356b1
I20250625 13:52:06.193454 1168 ts_tablet_manager.cc:1918] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:06.193574 1168 log.cc:1199] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/41a7862601f04fd7a938f7e5d5a4c0f9
I20250625 13:52:06.193912 1168 ts_tablet_manager.cc:1939] T 41a7862601f04fd7a938f7e5d5a4c0f9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:06.194056 1821 raft_consensus.cc:2802] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:06.194341 1821 raft_consensus.cc:695] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:06.194725 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 423) took 6575 ms. Trace:
I20250625 13:52:06.194813 1168 rpcz_store.cc:276] 0625 13:51:59.619296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.619374 (+ 78us) service_pool.cc:225] Handling call
0625 13:52:06.019537 (+6400163us) ts_tablet_manager.cc:1145] Deleting tablet 41a7862601f04fd7a938f7e5d5a4c0f9
0625 13:52:06.019544 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.126337 (+106793us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.193359 (+ 67022us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.194706 (+ 1347us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31537,"tablet-delete.queue_time_us":6399988}
I20250625 13:52:06.195214 1821 consensus_queue.cc:237] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.195298 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 41a7862601f04fd7a938f7e5d5a4c0f9 (table default.Table_478b0037c80e40048d4f8ebc92748f34 [id=13d38d05aad34e22884b9e07610b0e87]) successfully deleted
I20250625 13:52:06.200423 1165 ts_tablet_manager.cc:1918] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.200893 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 71170aaf26ed4f1b8c458d90db6e2765 (DEFAULT_TABLE table=default.Table_931f25deb4904d2fa74ab9437e733369 [id=36b6dd7a254247679b0eb5420e25a376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.201745 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71170aaf26ed4f1b8c458d90db6e2765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.204373 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 424) took 6545 ms. Trace:
I20250625 13:52:06.204643 1710 ts_tablet_manager.cc:1939] T 01ba67b500b646d7aa55315f7ee23ed6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.205026 889 tablet_service.cc:1468] Processing CreateTablet for tablet d5cb3b35a5c2483fa82226133a7c161d (DEFAULT_TABLE table=default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.205400 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cb3b35a5c2483fa82226133a7c161d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.205974 1710 tablet_replica.cc:331] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.202574 743 tablet_service.cc:1468] Processing CreateTablet for tablet 9302ac10a29c4b0c82268eb767615989 (DEFAULT_TABLE table=default.Table_734235bd5dc045678fdad3d130384a52 [id=f5be407389634e8393913e3b501f7b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.206449 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 01ba67b500b646d7aa55315f7ee23ed6 (table default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]) successfully deleted
W20250625 13:52:06.206702 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d8985f259eb54ffdab29a9f6a2c5ae16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8985f259eb54ffdab29a9f6a2c5ae16 already in progress: creating tablet
I20250625 13:52:06.206733 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9302ac10a29c4b0c82268eb767615989. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:06.206918 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet cab4329d58404014833ae5029f8f296d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cab4329d58404014833ae5029f8f296d already in progress: creating tablet
I20250625 13:52:06.204152 741 tablet_service.cc:1468] Processing CreateTablet for tablet d5cb3b35a5c2483fa82226133a7c161d (DEFAULT_TABLE table=default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.207381 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cb3b35a5c2483fa82226133a7c161d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.211444 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 9302ac10a29c4b0c82268eb767615989 (DEFAULT_TABLE table=default.Table_734235bd5dc045678fdad3d130384a52 [id=f5be407389634e8393913e3b501f7b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.211957 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9302ac10a29c4b0c82268eb767615989. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.213340 1165 log.cc:1199] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/af05fba6fe2042aab5dd47b30e4dfcf3
I20250625 13:52:06.213953 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9302ac10a29c4b0c82268eb767615989 (DEFAULT_TABLE table=default.Table_734235bd5dc045678fdad3d130384a52 [id=f5be407389634e8393913e3b501f7b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.214581 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ab516897fdf84efa9c63d6f43223d516"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.215173 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab516897fdf84efa9c63d6f43223d516 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.204476 1168 rpcz_store.cc:276] 0625 13:51:59.658963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.659020 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:06.204238 (+6545218us) ts_tablet_manager.cc:1145] Deleting tablet 77c50e1ebb9042ef8f0a651ba7186b74
0625 13:52:06.204248 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.204353 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6545002}
W20250625 13:52:06.215708 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 77c50e1ebb9042ef8f0a651ba7186b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77c50e1ebb9042ef8f0a651ba7186b74 already in progress: creating tablet
I20250625 13:52:06.216220 1165 ts_tablet_manager.cc:1939] T af05fba6fe2042aab5dd47b30e4dfcf3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:06.220418 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet af05fba6fe2042aab5dd47b30e4dfcf3 (table default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]) successfully deleted
I20250625 13:52:06.226055 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 529c3bf8e72f46948cfb864445ff51f3 (DEFAULT_TABLE table=default.Table_d0ce3bb2d8b04255ae3bc517960db379 [id=5227457618334dd0ad007356f6a79539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.226501 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529c3bf8e72f46948cfb864445ff51f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:06.228108 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5f49c0fc97d44b28acfd6cb493906edb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f49c0fc97d44b28acfd6cb493906edb already in progress: creating tablet
I20250625 13:52:06.228387 1165 tablet_replica.cc:331] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:52:06.228389 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d7c4a6a4fd574012818f3aa0a8b4d71f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c4a6a4fd574012818f3aa0a8b4d71f already in progress: creating tablet
W20250625 13:52:06.228912 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17efe97d559a45dc9c0ca0b74efa7aa6 already in progress: creating tablet
W20250625 13:52:06.229127 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 614181c6e4d24e7b9e5da3b4818cf065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 614181c6e4d24e7b9e5da3b4818cf065 already in progress: creating tablet
W20250625 13:52:06.229323 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e147e09b5244a8b8f52e8d10c3db4c1 already in progress: creating tablet
I20250625 13:52:06.232678 1710 raft_consensus.cc:2241] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:06.233125 1710 pending_rounds.cc:70] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:06.233219 1710 pending_rounds.cc:77] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520619299512320 op_type: NO_OP noop_request { }
I20250625 13:52:06.233435 1710 raft_consensus.cc:2887] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:06.233573 1710 raft_consensus.cc:2270] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.234114 1165 raft_consensus.cc:2241] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.234588 1165 raft_consensus.cc:2270] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:06.233207 1949 consensus_queue.cc:1162] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250625 13:52:06.235656 1165 ts_tablet_manager.cc:1905] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.236164 1710 ts_tablet_manager.cc:1905] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.236917 1837 raft_consensus.cc:491] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.237915 1776 raft_consensus.cc:491] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.238040 1776 raft_consensus.cc:513] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.238608 1776 leader_election.cc:290] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:06.239468 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d95a376304e464ab75e2e1e00e7a924" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.240020 711 leader_election.cc:343] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.243432 1710 ts_tablet_manager.cc:1918] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:06.243665 1710 log.cc:1199] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ae4a105e80c1460cbb21e1bdb9b356b1
I20250625 13:52:06.244895 1710 ts_tablet_manager.cc:1939] T ae4a105e80c1460cbb21e1bdb9b356b1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.245148 1842 raft_consensus.cc:491] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.245172 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_66186ab6312345c1a94ca39ad501c296" } new_table_name: "default.Table_330a9a9673154b0fb697d5f2cc16a4ab" modify_external_catalogs: true
I20250625 13:52:06.246094 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ae4a105e80c1460cbb21e1bdb9b356b1 (table default.Table_d18abaf5b02340fa82d3fee69699e1e3 [id=ae6ea457ce8e4c7eb7c5e7c1a9d62c7a]) successfully deleted
I20250625 13:52:06.247694 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
W20250625 13:52:06.248067 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d3a17198b9a04138b24022cf8506ab36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a17198b9a04138b24022cf8506ab36 already in progress: creating tablet
W20250625 13:52:06.248806 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae4b428ae7bc49fcbc37e9d7d238ec21 already in progress: creating tablet
I20250625 13:52:06.249577 590 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet c7c3f0ca6fa5482db7f57143911c838f (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]). Expected version 6 got 0
W20250625 13:52:06.249578 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 43941422259b4413bc71acfce8928e1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43941422259b4413bc71acfce8928e1e already in progress: creating tablet
W20250625 13:52:06.250346 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b4be0bc23cde4f0f94bd678096e0c7d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4be0bc23cde4f0f94bd678096e0c7d8 already in progress: creating tablet
W20250625 13:52:06.251686 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 43b791fd11914591bb812cc155d950dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b791fd11914591bb812cc155d950dc already in progress: creating tablet
I20250625 13:52:06.252408 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d5cb3b35a5c2483fa82226133a7c161d (DEFAULT_TABLE table=default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.253005 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cb3b35a5c2483fa82226133a7c161d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.253767 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 22bcbfcfeb0a439288622994b2d245b2 (DEFAULT_TABLE table=default.Table_cea4a53bfa854372839e62197285c18c [id=1f771e558b7f44f1a12f33b823d1e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.253945 1019 tablet_service.cc:1468] Processing CreateTablet for tablet c366da4fb138453386d97fba0f5c91be (DEFAULT_TABLE table=default.Table_57e8a917a4764270a0b6ef4a4cd53dc8 [id=ec3dfab61fff4a9ebb227df0e8816a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.254231 741 tablet_service.cc:1468] Processing CreateTablet for tablet 22bcbfcfeb0a439288622994b2d245b2 (DEFAULT_TABLE table=default.Table_cea4a53bfa854372839e62197285c18c [id=1f771e558b7f44f1a12f33b823d1e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.254599 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bcbfcfeb0a439288622994b2d245b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.254895 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bcbfcfeb0a439288622994b2d245b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.259577 757 tablet_service.cc:1468] Processing CreateTablet for tablet c366da4fb138453386d97fba0f5c91be (DEFAULT_TABLE table=default.Table_57e8a917a4764270a0b6ef4a4cd53dc8 [id=ec3dfab61fff4a9ebb227df0e8816a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.259971 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c366da4fb138453386d97fba0f5c91be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.261595 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c366da4fb138453386d97fba0f5c91be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.262656 1165 ts_tablet_manager.cc:1918] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.262771 1165 log.cc:1199] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/36b151e200e1430c99ac194bc55881c1
I20250625 13:52:06.263070 1165 ts_tablet_manager.cc:1939] T 36b151e200e1430c99ac194bc55881c1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:06.263746 1842 raft_consensus.cc:513] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.264006 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 36b151e200e1430c99ac194bc55881c1 (table default.Table_29924a11586f4d2bb3d0f8574a7849f8 [id=e057b2ec6f914a01b474481bc5c25349]) successfully deleted
I20250625 13:52:06.264425 1842 leader_election.cc:290] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.237062 1837 raft_consensus.cc:513] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.265141 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0a3ea18bab944749d45da9b3139f8bc" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.265420 851 leader_election.cc:343] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: c0a3ea18bab944749d45da9b3139f8bc
W20250625 13:52:06.265616 849 leader_election.cc:343] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.265714 849 leader_election.cc:304] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.265792 1168 tablet_replica.cc:331] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:06.265977 1168 raft_consensus.cc:2241] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.266076 1842 raft_consensus.cc:2747] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.266117 1168 raft_consensus.cc:2270] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.266637 1168 ts_tablet_manager.cc:1905] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.267367 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9302ac10a29c4b0c82268eb767615989. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.265758 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d95a376304e464ab75e2e1e00e7a924" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:06.268193 711 leader_election.cc:343] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.268319 711 leader_election.cc:304] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:06.268682 1776 raft_consensus.cc:2747] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.263571 892 tablet_service.cc:1468] Processing CreateTablet for tablet c366da4fb138453386d97fba0f5c91be (DEFAULT_TABLE table=default.Table_57e8a917a4764270a0b6ef4a4cd53dc8 [id=ec3dfab61fff4a9ebb227df0e8816a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.270315 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "637847cac1724bfeb629c4d4013396ca" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:06.271504 1837 leader_election.cc:290] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:06.272250 851 leader_election.cc:343] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 637847cac1724bfeb629c4d4013396ca
I20250625 13:52:06.274015 1171 tablet_bootstrap.cc:492] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:06.274152 1171 ts_tablet_manager.cc:1397] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.206s user 0.001s sys 0.003s
W20250625 13:52:06.274746 849 leader_election.cc:343] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.274839 849 leader_election.cc:304] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:06.275110 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet af05fba6fe2042aab5dd47b30e4dfcf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af05fba6fe2042aab5dd47b30e4dfcf3 already in progress: creating tablet
I20250625 13:52:06.275298 1837 raft_consensus.cc:2747] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.277598 1710 tablet_replica.cc:331] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.277565 1171 raft_consensus.cc:357] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.277746 1171 raft_consensus.cc:383] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.277796 1171 raft_consensus.cc:738] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.277995 1171 consensus_queue.cc:260] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.278475 1710 raft_consensus.cc:2241] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.278517 1171 ts_tablet_manager.cc:1428] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:06.279918 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c366da4fb138453386d97fba0f5c91be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.280184 1710 raft_consensus.cc:2270] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.280429 1171 tablet_bootstrap.cc:492] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:06.281284 1710 ts_tablet_manager.cc:1905] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.291239 1170 tablet_bootstrap.cc:492] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.291507 1170 ts_tablet_manager.cc:1397] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.184s user 0.009s sys 0.009s
I20250625 13:52:06.292348 1170 raft_consensus.cc:357] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.292578 1170 raft_consensus.cc:383] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.292743 1170 raft_consensus.cc:738] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.293051 1170 consensus_queue.cc:260] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.295192 1170 ts_tablet_manager.cc:1428] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250625 13:52:06.295540 1170 tablet_bootstrap.cc:492] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.297674 1170 tablet_bootstrap.cc:654] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.297940 1171 tablet_bootstrap.cc:654] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.300902 1710 ts_tablet_manager.cc:1918] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.301805 1710 log.cc:1199] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/6771d8e180fe48569a0e016db0cc9b51
I20250625 13:52:06.302135 1710 ts_tablet_manager.cc:1939] T 6771d8e180fe48569a0e016db0cc9b51 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.303148 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 6771d8e180fe48569a0e016db0cc9b51 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]) successfully deleted
W20250625 13:52:06.304066 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5f49c0fc97d44b28acfd6cb493906edb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f49c0fc97d44b28acfd6cb493906edb already in progress: creating tablet
W20250625 13:52:06.304919 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d7c4a6a4fd574012818f3aa0a8b4d71f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c4a6a4fd574012818f3aa0a8b4d71f already in progress: creating tablet
W20250625 13:52:06.305660 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17efe97d559a45dc9c0ca0b74efa7aa6 already in progress: creating tablet
W20250625 13:52:06.306382 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 614181c6e4d24e7b9e5da3b4818cf065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 614181c6e4d24e7b9e5da3b4818cf065 already in progress: creating tablet
I20250625 13:52:06.306836 1710 tablet_replica.cc:331] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.307180 1710 raft_consensus.cc:2241] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.307548 1710 raft_consensus.cc:2270] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.308487 1710 ts_tablet_manager.cc:1905] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.313714 1170 tablet_bootstrap.cc:492] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.313889 1170 ts_tablet_manager.cc:1397] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20250625 13:52:06.314729 1170 raft_consensus.cc:357] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.314975 1170 raft_consensus.cc:383] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.315073 1170 raft_consensus.cc:738] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.315359 1170 consensus_queue.cc:260] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.317524 1170 ts_tablet_manager.cc:1428] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:06.317840 1170 tablet_bootstrap.cc:492] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.320295 1170 tablet_bootstrap.cc:654] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.320255 1842 raft_consensus.cc:491] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.320398 1842 raft_consensus.cc:513] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.329768 1170 tablet_bootstrap.cc:492] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.329938 1842 leader_election.cc:290] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.329983 1170 ts_tablet_manager.cc:1397] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20250625 13:52:06.330552 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af05fba6fe2042aab5dd47b30e4dfcf3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:06.331023 1170 raft_consensus.cc:357] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.331244 1170 raft_consensus.cc:383] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:52:06.331277 849 leader_election.cc:343] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.331379 1170 raft_consensus.cc:738] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.331771 1170 consensus_queue.cc:260] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:06.332087 851 leader_election.cc:343] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: af05fba6fe2042aab5dd47b30e4dfcf3
I20250625 13:52:06.332186 851 leader_election.cc:304] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.332656 1842 raft_consensus.cc:2747] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.334254 1170 ts_tablet_manager.cc:1428] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:06.334583 1170 tablet_bootstrap.cc:492] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.326303 1710 ts_tablet_manager.cc:1918] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:06.340029 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc75a0a8421b457089cc707d9eb6b76f consensus state has no leader
I20250625 13:52:06.340216 1170 tablet_bootstrap.cc:654] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.342674 881 tablet_service.cc:1468] Processing CreateTablet for tablet 22bcbfcfeb0a439288622994b2d245b2 (DEFAULT_TABLE table=default.Table_cea4a53bfa854372839e62197285c18c [id=1f771e558b7f44f1a12f33b823d1e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.343041 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bcbfcfeb0a439288622994b2d245b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.343948 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3ade196aa4334cff912b6e96e532483c" } new_table_name: "default.Table_285a51f0c5a84535bfe43c3a58a8e295" modify_external_catalogs: true
I20250625 13:52:06.355268 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 44c25e1681d146e393fa3bcc38acb9a0 (DEFAULT_TABLE table=default.Table_abea1e01b70d4edeb3cfbbd851205b11 [id=5a5d1aacd01f4735b5b5d147238aa919]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.355798 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c25e1681d146e393fa3bcc38acb9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.370048 1776 raft_consensus.cc:491] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.370223 1776 raft_consensus.cc:513] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.370823 1776 leader_election.cc:290] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:06.371534 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9267df231f354c578c38f452418e523c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.372042 711 leader_election.cc:343] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.372581 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9267df231f354c578c38f452418e523c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:06.373131 711 leader_election.cc:343] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.373267 711 leader_election.cc:304] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:06.373713 1776 raft_consensus.cc:2747] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.375442 743 tablet_service.cc:1468] Processing CreateTablet for tablet 44c25e1681d146e393fa3bcc38acb9a0 (DEFAULT_TABLE table=default.Table_abea1e01b70d4edeb3cfbbd851205b11 [id=5a5d1aacd01f4735b5b5d147238aa919]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.375823 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c25e1681d146e393fa3bcc38acb9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.206077 1166 tablet_bootstrap.cc:492] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.375864 1710 log.cc:1199] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/5d7704b38144429b9c95675be84a4730
I20250625 13:52:06.376008 1166 ts_tablet_manager.cc:1397] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.189s user 0.003s sys 0.000s
I20250625 13:52:06.376211 1710 ts_tablet_manager.cc:1939] T 5d7704b38144429b9c95675be84a4730 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.376792 1166 raft_consensus.cc:357] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.376937 1166 raft_consensus.cc:383] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.376984 1166 raft_consensus.cc:738] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.380553 1168 ts_tablet_manager.cc:1918] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.380678 1168 log.cc:1199] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/7d361e73fe3d4ea2bb6735f814345ebb
I20250625 13:52:06.382968 741 tablet_service.cc:1468] Processing CreateTablet for tablet 1c4a6d7d0c9647a4a17dca28d6a063a5 (DEFAULT_TABLE table=default.Table_3f882e0fab474cf2aef02b0067d704f8 [id=515860fa91ae43bea4530116b0306433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.384697 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 5d7704b38144429b9c95675be84a4730 (table default.Table_8c4176baf42b43658ec245c90d86cb98 [id=1786625fb4994b51a8b90c53e7e1714c]) successfully deleted
I20250625 13:52:06.385604 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 1c4a6d7d0c9647a4a17dca28d6a063a5 (DEFAULT_TABLE table=default.Table_3f882e0fab474cf2aef02b0067d704f8 [id=515860fa91ae43bea4530116b0306433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.386072 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c4a6d7d0c9647a4a17dca28d6a063a5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:06.387552 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e147e09b5244a8b8f52e8d10c3db4c1 already in progress: creating tablet
I20250625 13:52:06.392884 893 tablet_service.cc:1468] Processing CreateTablet for tablet 1c4a6d7d0c9647a4a17dca28d6a063a5 (DEFAULT_TABLE table=default.Table_3f882e0fab474cf2aef02b0067d704f8 [id=515860fa91ae43bea4530116b0306433]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.393317 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c4a6d7d0c9647a4a17dca28d6a063a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.393669 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c4a6d7d0c9647a4a17dca28d6a063a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.395416 1166 consensus_queue.cc:260] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.397135 1776 raft_consensus.cc:491] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.397253 1776 raft_consensus.cc:513] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.398097 1776 leader_election.cc:290] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:06.399046 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b0037d09796455cbaaadfaa9c010673" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.399585 711 leader_election.cc:343] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.401506 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3bf17a9252fa4b4d9150da868f39d897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.401353 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:52:06.401808 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:52:06.401927 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:52:06.402169 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bf17a9252fa4b4d9150da868f39d897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.406919 1166 ts_tablet_manager.cc:1428] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20250625 13:52:06.408751 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b584197acb904919a3681da75860d445"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.409475 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b584197acb904919a3681da75860d445 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.413471 1166 tablet_bootstrap.cc:492] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.415532 1166 tablet_bootstrap.cc:654] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.420964 1170 tablet_bootstrap.cc:492] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.421103 1170 ts_tablet_manager.cc:1397] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.087s user 0.006s sys 0.019s
I20250625 13:52:06.421820 1170 raft_consensus.cc:357] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.422006 1170 raft_consensus.cc:383] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.422052 1170 raft_consensus.cc:738] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.422245 1170 consensus_queue.cc:260] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.422683 1170 ts_tablet_manager.cc:1428] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:06.422977 1170 tablet_bootstrap.cc:492] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:06.424849 849 consensus_peers.cc:487] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1acc41a93f6433588727811903921fe. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:06.426779 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b0037d09796455cbaaadfaa9c010673" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:06.427837 711 leader_election.cc:343] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.427963 711 leader_election.cc:304] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:06.428424 1842 raft_consensus.cc:491] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.428532 1842 raft_consensus.cc:513] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.429032 1842 leader_election.cc:290] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.429309 1776 raft_consensus.cc:2747] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.429926 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a22ade8cefac4140b08882c311ea5181" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:06.430353 1170 tablet_bootstrap.cc:654] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:06.430558 849 leader_election.cc:343] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:06.430661 851 leader_election.cc:343] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a22ade8cefac4140b08882c311ea5181
I20250625 13:52:06.430748 851 leader_election.cc:304] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.375698 889 tablet_service.cc:1468] Processing CreateTablet for tablet 44c25e1681d146e393fa3bcc38acb9a0 (DEFAULT_TABLE table=default.Table_abea1e01b70d4edeb3cfbbd851205b11 [id=5a5d1aacd01f4735b5b5d147238aa919]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.430934 1842 raft_consensus.cc:2747] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.431435 1168 ts_tablet_manager.cc:1939] T 7d361e73fe3d4ea2bb6735f814345ebb P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:06.436229 1794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:52:06.436345 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:52:06.436447 1794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:52:06.436514 1794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:52:06.437060 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c25e1681d146e393fa3bcc38acb9a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.439841 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 428) took 6659 ms. Trace:
I20250625 13:52:06.440259 1168 rpcz_store.cc:276] 0625 13:51:59.780155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.780218 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:06.216569 (+6436351us) ts_tablet_manager.cc:1145] Deleting tablet 7d361e73fe3d4ea2bb6735f814345ebb
0625 13:52:06.265758 (+ 49189us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.308332 (+ 42574us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.341471 (+ 33139us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.439816 (+ 98345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6436102}
I20250625 13:52:06.441058 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 7d361e73fe3d4ea2bb6735f814345ebb (table default.Table_015b4a77da244eaf9d34a281215e77c7 [id=2603504860f8462b90699ba1f6f599d7]) successfully deleted
I20250625 13:52:06.442940 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 429) took 6623 ms. Trace:
I20250625 13:52:06.443116 1168 rpcz_store.cc:276] 0625 13:51:59.819171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.819230 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:06.441399 (+6622169us) ts_tablet_manager.cc:1145] Deleting tablet 4c9efc7631df4ad29112a41f229ae2e9
0625 13:52:06.441409 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.442905 (+ 1496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6619034}
W20250625 13:52:06.443980 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4c9efc7631df4ad29112a41f229ae2e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c9efc7631df4ad29112a41f229ae2e9 already in progress: creating tablet
I20250625 13:52:06.444525 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 430) took 6620 ms. Trace:
I20250625 13:52:06.444670 1168 rpcz_store.cc:276] 0625 13:51:59.824345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.824406 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:06.444378 (+6619972us) ts_tablet_manager.cc:1145] Deleting tablet bb648c45ea344fb3bac765c9e06db2b1
0625 13:52:06.444392 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.444501 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6619682}
W20250625 13:52:06.445271 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bb648c45ea344fb3bac765c9e06db2b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb648c45ea344fb3bac765c9e06db2b1 already in progress: creating tablet
I20250625 13:52:06.445673 1168 tablet_replica.cc:331] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:06.445981 1168 raft_consensus.cc:2241] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.446235 1168 raft_consensus.cc:2270] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:06.451094 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e99fe5ad690049029b8325dd38a11580 consensus state has no leader
I20250625 13:52:06.451365 594 master_service.cc:948] ReplaceTablet: received request to replace tablet a028df315713443986c4b362ed17fd99 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:06.452481 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 8e2bc4df39674dc3be0cd32c68f1ebca from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:06.453420 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35c7bf4272994e9b8a59062b1757f6f5 has no consensus state
I20250625 13:52:06.459403 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_295dd4f121234e659090b8e4cf70faee" } modify_external_catalogs: true
I20250625 13:52:06.459647 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_295dd4f121234e659090b8e4cf70faee" } modify_external_catalogs: true
I20250625 13:52:06.462477 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4ecc6555bb17447bb819b95c6b928c7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.463160 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ecc6555bb17447bb819b95c6b928c7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.464689 588 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1] alter complete (version 6)
I20250625 13:52:06.465354 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d18abaf5b02340fa82d3fee69699e1e3" } modify_external_catalogs: true
I20250625 13:52:06.465509 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d18abaf5b02340fa82d3fee69699e1e3" } modify_external_catalogs: true
I20250625 13:52:06.467161 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 68a7712455694a1cbead413230620827 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:06.469537 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_57e8a917a4764270a0b6ef4a4cd53dc8" } modify_external_catalogs: true
I20250625 13:52:06.469767 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_57e8a917a4764270a0b6ef4a4cd53dc8" } modify_external_catalogs: true
I20250625 13:52:06.469898 1794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6764
ASAN enabled
I20250625 13:52:06.471791 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a31f10c8ca745b7ad635c622a265450 with delete_type TABLET_DATA_DELETED (replaced by tablet df1ed922750f45eab02d1fc0c26af75e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.472102 1710 tablet_replica.cc:331] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.472270 1710 raft_consensus.cc:2241] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.472417 1710 raft_consensus.cc:2270] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.473183 1710 ts_tablet_manager.cc:1905] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:06.473945 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07cfdcf3e8b74c24b48b62bdc8bde971 consensus state has no leader
I20250625 13:52:06.476243 1794 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:52:06.477276 884 tablet_service.cc:1515] Processing DeleteTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.480600 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f00e72e103f443e886f0df690cff4b6f with delete_type TABLET_DATA_DELETED (replaced by tablet a0e81d4c904248eba858462bcf31c5d9) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.481719 1710 ts_tablet_manager.cc:1918] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.481839 1710 log.cc:1199] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/9a31f10c8ca745b7ad635c622a265450
I20250625 13:52:06.482128 1710 ts_tablet_manager.cc:1939] T 9a31f10c8ca745b7ad635c622a265450 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.478920 1794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 13:52:06.483196 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 9a31f10c8ca745b7ad635c622a265450 (table default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]) successfully deleted
I20250625 13:52:06.483530 1710 tablet_replica.cc:331] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.484076 1710 raft_consensus.cc:2241] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.484282 1710 raft_consensus.cc:2270] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.484465 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:06.484999 1710 ts_tablet_manager.cc:1905] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.485560 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_25d9c73fdf1b43b787345769f54915d6" } modify_external_catalogs: true
I20250625 13:52:06.485793 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_25d9c73fdf1b43b787345769f54915d6" } modify_external_catalogs: true
I20250625 13:52:06.487990 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b0037d09796455cbaaadfaa9c010673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.488267 1165 tablet_replica.cc:331] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:06.488500 1165 raft_consensus.cc:2241] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.488685 1165 raft_consensus.cc:2270] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.489533 1165 ts_tablet_manager.cc:1905] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.495700 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ab663b7eba452c9f9775c0f568010b with delete_type TABLET_DATA_DELETED (replaced by tablet 155737a8d5604eb69524c4d6bea61550) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.495930 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 54fa042947da4e1b89018640b46b271a with delete_type TABLET_DATA_DELETED (replaced by tablet ab7202a8761444a0a731993d890fcdeb) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.498549 1776 raft_consensus.cc:491] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.498682 1776 raft_consensus.cc:513] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.499234 1776 leader_election.cc:290] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:06.499863 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc75a0a8421b457089cc707d9eb6b76f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:06.500414 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc75a0a8421b457089cc707d9eb6b76f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:06.500860 1710 ts_tablet_manager.cc:1918] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.500957 1710 log.cc:1199] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/f00e72e103f443e886f0df690cff4b6f
W20250625 13:52:06.500945 711 leader_election.cc:343] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.501183 1165 ts_tablet_manager.cc:1918] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.501233 1710 ts_tablet_manager.cc:1939] T f00e72e103f443e886f0df690cff4b6f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:06.501288 711 leader_election.cc:343] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.501435 711 leader_election.cc:304] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:06.501876 1776 raft_consensus.cc:2747] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.502157 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet f00e72e103f443e886f0df690cff4b6f (table default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]) successfully deleted
I20250625 13:52:06.502468 1710 tablet_replica.cc:331] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.502636 1710 raft_consensus.cc:2241] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.502787 1710 raft_consensus.cc:2270] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.503511 1710 ts_tablet_manager.cc:1905] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.504040 1171 tablet_bootstrap.cc:492] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:06.504155 1171 ts_tablet_manager.cc:1397] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.224s user 0.016s sys 0.024s
I20250625 13:52:06.504927 1171 raft_consensus.cc:357] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.504990 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 46316eb725bc47e695af476c0b185504 with delete_type TABLET_DATA_DELETED (replaced by tablet b7d888e628a94f939ecf2571e60b1948) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.505076 1171 raft_consensus.cc:383] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.505131 1171 raft_consensus.cc:738] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.501291 1165 log.cc:1199] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/3b0037d09796455cbaaadfaa9c010673
I20250625 13:52:06.505311 1171 consensus_queue.cc:260] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.505499 1165 ts_tablet_manager.cc:1939] T 3b0037d09796455cbaaadfaa9c010673 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:06.506662 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 3b0037d09796455cbaaadfaa9c010673 (table default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae [id=6f1fa6d1db4e4abaae4d06299152224f]) successfully deleted
I20250625 13:52:06.508064 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 6e8b9b732f8948d2b4c1a81a1de4489a from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:06.513273 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cd5303e674246ada7d36574b535b333 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.515905 1710 ts_tablet_manager.cc:1918] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.516080 1710 log.cc:1199] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/34ab663b7eba452c9f9775c0f568010b
I20250625 13:52:06.516400 1710 ts_tablet_manager.cc:1939] T 34ab663b7eba452c9f9775c0f568010b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.517761 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 34ab663b7eba452c9f9775c0f568010b (table default.Table_6a4abf185c2b44d3ba20510778eead0a [id=c2e58d3856d7401ca7b97d9cce92bbce]) successfully deleted
I20250625 13:52:06.518122 1710 tablet_replica.cc:331] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.518325 1710 raft_consensus.cc:2241] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.518544 1710 raft_consensus.cc:2270] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.519291 1710 ts_tablet_manager.cc:1905] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.527577 1710 ts_tablet_manager.cc:1918] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.527792 1710 log.cc:1199] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/54fa042947da4e1b89018640b46b271a
I20250625 13:52:06.528113 1710 ts_tablet_manager.cc:1939] T 54fa042947da4e1b89018640b46b271a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:06.528622 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 35c7bf4272994e9b8a59062b1757f6f5 (table default.Table_e5270cb847db4a57979fd35c13f6fe0b [id=607b89f622c240f0917158cf08a94f81], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 35c7bf4272994e9b8a59062b1757f6f5 has no consensus state
I20250625 13:52:06.529500 1710 tablet_replica.cc:331] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.529781 1710 raft_consensus.cc:2241] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.529963 1710 raft_consensus.cc:2270] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.530709 1710 ts_tablet_manager.cc:1905] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.532091 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a53e9e99051944a1bc66244b53112731" } modify_external_catalogs: true
I20250625 13:52:06.532434 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a53e9e99051944a1bc66244b53112731" } modify_external_catalogs: true
I20250625 13:52:06.532727 1168 ts_tablet_manager.cc:1905] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.536722 1710 ts_tablet_manager.cc:1918] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.536839 1710 log.cc:1199] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/46316eb725bc47e695af476c0b185504
I20250625 13:52:06.537111 1710 ts_tablet_manager.cc:1939] T 46316eb725bc47e695af476c0b185504 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.541829 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a028df315713443986c4b362ed17fd99
I20250625 13:52:06.542179 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 54fa042947da4e1b89018640b46b271a (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]) successfully deleted
I20250625 13:52:06.542378 594 catalog_manager.cc:6652] ReplaceTablet: tablet a028df315713443986c4b362ed17fd99 deleted and replaced by tablet c9ae350a9dbf476081dc56a709de42c8
I20250625 13:52:06.542430 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 46316eb725bc47e695af476c0b185504 (table default.Table_7a2ca389ecd148b9be0e90c130fe6431 [id=26d0bdba8be543d89d5337c86bbf6e57]) successfully deleted
I20250625 13:52:06.542613 594 master_service.cc:948] ReplaceTablet: received request to replace tablet cbe4046742b34f1ab0b1610b5d6ee01e from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:06.542954 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet a028df315713443986c4b362ed17fd99 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ae350a9dbf476081dc56a709de42c8) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.543252 1166 tablet_bootstrap.cc:492] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.543422 1166 ts_tablet_manager.cc:1397] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.130s user 0.004s sys 0.000s
I20250625 13:52:06.543570 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a028df315713443986c4b362ed17fd99 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ae350a9dbf476081dc56a709de42c8) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.544597 1170 tablet_bootstrap.cc:492] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.544726 1170 ts_tablet_manager.cc:1397] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.122s user 0.025s sys 0.030s
I20250625 13:52:06.544807 757 tablet_service.cc:1515] Processing DeleteTablet for tablet a028df315713443986c4b362ed17fd99 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ae350a9dbf476081dc56a709de42c8) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:06.545029 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a028df315713443986c4b362ed17fd99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a028df315713443986c4b362ed17fd99 already in progress: creating tablet
I20250625 13:52:06.545030 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b14d00c93a964edd815122e3723f89e7
I20250625 13:52:06.544234 1166 raft_consensus.cc:357] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.545253 1166 raft_consensus.cc:383] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.545318 1166 raft_consensus.cc:738] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.545368 1170 raft_consensus.cc:357] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.545521 1170 raft_consensus.cc:383] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.545585 1170 raft_consensus.cc:738] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.545547 1166 consensus_queue.cc:260] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.545949 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f766c58c1858460cbad7a7f98f1ac55e" } new_table_name: "default.Table_764d5efe3d33490a8c0619e033673fb7" modify_external_catalogs: true
I20250625 13:52:06.545804 1170 consensus_queue.cc:260] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.549409 1170 ts_tablet_manager.cc:1428] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20250625 13:52:06.549690 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a028df315713443986c4b362ed17fd99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a028df315713443986c4b362ed17fd99 already in progress: creating tablet
I20250625 13:52:06.550150 1170 tablet_bootstrap.cc:492] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.551067 1166 ts_tablet_manager.cc:1428] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 13:52:06.551658 741 tablet_service.cc:1515] Processing DeleteTablet for tablet b14d00c93a964edd815122e3723f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.552325 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet b14d00c93a964edd815122e3723f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:06.552320 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet b14d00c93a964edd815122e3723f89e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b14d00c93a964edd815122e3723f89e7 already in progress: creating tablet
I20250625 13:52:06.552632 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b14d00c93a964edd815122e3723f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:06.553642 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b14d00c93a964edd815122e3723f89e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b14d00c93a964edd815122e3723f89e7 already in progress: creating tablet
I20250625 13:52:06.553959 1166 tablet_bootstrap.cc:492] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.555583 1168 ts_tablet_manager.cc:1918] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.555696 1168 log.cc:1199] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/dbd2120b4b084691b6604305773e74f9
I20250625 13:52:06.556000 1168 ts_tablet_manager.cc:1939] T dbd2120b4b084691b6604305773e74f9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:06.556712 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 431) took 6711 ms. Trace:
I20250625 13:52:06.556749 1170 tablet_bootstrap.cc:654] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.556809 1168 rpcz_store.cc:276] 0625 13:51:59.845550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.845603 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:06.445616 (+6600013us) ts_tablet_manager.cc:1145] Deleting tablet dbd2120b4b084691b6604305773e74f9
0625 13:52:06.445629 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.536436 (+ 90807us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.555463 (+ 19027us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.556693 (+ 1230us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6599820}
I20250625 13:52:06.557839 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet dbd2120b4b084691b6604305773e74f9 (table default.Table_8910e07e05ad4d71bfff7d5a0d6952f4 [id=2dd7357a611c4866913dfd01fedcbe5b]) successfully deleted
I20250625 13:52:06.558321 1166 tablet_bootstrap.cc:654] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.564838 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8e2bc4df39674dc3be0cd32c68f1ebca
I20250625 13:52:06.565465 597 catalog_manager.cc:6652] ReplaceTablet: tablet 8e2bc4df39674dc3be0cd32c68f1ebca deleted and replaced by tablet 67e05cc542384b4b8117a1d9330db24b
I20250625 13:52:06.565891 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f5f3d8e484094d71808796036298fd34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.566625 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5f3d8e484094d71808796036298fd34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.567435 1166 tablet_bootstrap.cc:492] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.567584 1166 ts_tablet_manager.cc:1397] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20250625 13:52:06.568432 1166 raft_consensus.cc:357] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.569011 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca with delete_type TABLET_DATA_DELETED (replaced by tablet 67e05cc542384b4b8117a1d9330db24b) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.569366 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca with delete_type TABLET_DATA_DELETED (replaced by tablet 67e05cc542384b4b8117a1d9330db24b) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:06.570111 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8e2bc4df39674dc3be0cd32c68f1ebca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2bc4df39674dc3be0cd32c68f1ebca already in progress: creating tablet
I20250625 13:52:06.569535 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca with delete_type TABLET_DATA_DELETED (replaced by tablet 67e05cc542384b4b8117a1d9330db24b) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.570370 1166 raft_consensus.cc:383] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.570439 1166 raft_consensus.cc:738] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.569437 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b129117db064a15b27e82439637cf22 with delete_type TABLET_DATA_DELETED (replaced by tablet 35aedb3b590d4917bac8fe0962d68fb3) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.570695 1166 consensus_queue.cc:260] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.571002 1165 tablet_replica.cc:331] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:52:06.571137 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8e2bc4df39674dc3be0cd32c68f1ebca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2bc4df39674dc3be0cd32c68f1ebca already in progress: creating tablet
I20250625 13:52:06.571506 1166 ts_tablet_manager.cc:1428] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:06.571826 1165 raft_consensus.cc:2241] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.571976 1165 raft_consensus.cc:2270] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.572193 1166 tablet_bootstrap.cc:492] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.574025 1166 tablet_bootstrap.cc:654] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.574679 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b7062c34fd143bf86b743055a27db8c with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd84b8afa2548fe8ce5925f0bf806f0) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.574694 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 436) took 6612 ms. Trace:
I20250625 13:52:06.574774 1168 rpcz_store.cc:276] 0625 13:51:59.962504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:51:59.962576 (+ 72us) service_pool.cc:225] Handling call
0625 13:52:06.574533 (+6611957us) ts_tablet_manager.cc:1145] Deleting tablet dcf7fab53cae49908d2952cdaa07943a
0625 13:52:06.574546 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.574671 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6593695}
I20250625 13:52:06.575003 1710 tablet_replica.cc:331] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.575163 1710 raft_consensus.cc:2241] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:06.575167 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet dcf7fab53cae49908d2952cdaa07943a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcf7fab53cae49908d2952cdaa07943a already in progress: creating tablet
I20250625 13:52:06.575301 1710 raft_consensus.cc:2270] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.576399 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 447) took 6524 ms. Trace:
I20250625 13:52:06.576473 1168 rpcz_store.cc:276] 0625 13:52:00.051598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.052045 (+ 447us) service_pool.cc:225] Handling call
0625 13:52:06.576111 (+6524066us) ts_tablet_manager.cc:1145] Deleting tablet 077e8c001cc646559b5268d4a48c0ad6
0625 13:52:06.576282 (+ 171us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.576384 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6515665}
I20250625 13:52:06.576756 1171 ts_tablet_manager.cc:1428] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.072s user 0.018s sys 0.000s
W20250625 13:52:06.576886 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 077e8c001cc646559b5268d4a48c0ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 077e8c001cc646559b5268d4a48c0ad6 already in progress: creating tablet
I20250625 13:52:06.579954 741 tablet_service.cc:1515] Processing DeleteTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.580807 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3a79d0adb2204bcaa209ad1de0fa64fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.581367 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a79d0adb2204bcaa209ad1de0fa64fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.583616 1171 tablet_bootstrap.cc:492] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:06.585508 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 458) took 6337 ms. Trace:
I20250625 13:52:06.585597 1168 rpcz_store.cc:276] 0625 13:52:00.247628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.332141 (+ 84513us) service_pool.cc:225] Handling call
0625 13:52:06.585315 (+6253174us) ts_tablet_manager.cc:1145] Deleting tablet 7efc8df3c8e24020b073a1b61bc9e0a0
0625 13:52:06.585323 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.585493 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6252966}
W20250625 13:52:06.585968 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7efc8df3c8e24020b073a1b61bc9e0a0 already in progress: creating tablet
I20250625 13:52:06.586046 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 459) took 6338 ms. Trace:
I20250625 13:52:06.586123 1168 rpcz_store.cc:276] 0625 13:52:00.247752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.332367 (+ 84615us) service_pool.cc:225] Handling call
0625 13:52:06.585936 (+6253569us) ts_tablet_manager.cc:1145] Deleting tablet 829221b8a79f460d90dd24be6a2d4c43
0625 13:52:06.585944 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.586032 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6253438}
I20250625 13:52:06.586526 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 460) took 6338 ms. Trace:
I20250625 13:52:06.586609 1168 rpcz_store.cc:276] 0625 13:52:00.247829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.332506 (+ 84677us) service_pool.cc:225] Handling call
0625 13:52:06.586437 (+6253931us) ts_tablet_manager.cc:1145] Deleting tablet 150af49427ae47b98e9b2077bbbe1e2b
0625 13:52:06.586446 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.586512 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6253793}
W20250625 13:52:06.586705 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 829221b8a79f460d90dd24be6a2d4c43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829221b8a79f460d90dd24be6a2d4c43 already in progress: creating tablet
W20250625 13:52:06.586993 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 150af49427ae47b98e9b2077bbbe1e2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 150af49427ae47b98e9b2077bbbe1e2b already in progress: creating tablet
I20250625 13:52:06.587013 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 461) took 6339 ms. Trace:
I20250625 13:52:06.587080 1168 rpcz_store.cc:276] 0625 13:52:00.247908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.332649 (+ 84741us) service_pool.cc:225] Handling call
0625 13:52:06.586920 (+6254271us) ts_tablet_manager.cc:1145] Deleting tablet ff65e23c44624b2cbf92c2dc67eaed09
0625 13:52:06.586928 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.586998 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6254125}
I20250625 13:52:06.587435 1171 tablet_bootstrap.cc:654] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:06.587620 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ff65e23c44624b2cbf92c2dc67eaed09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff65e23c44624b2cbf92c2dc67eaed09 already in progress: creating tablet
I20250625 13:52:06.588830 1166 tablet_bootstrap.cc:492] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.588958 1166 ts_tablet_manager.cc:1397] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250625 13:52:06.590044 1166 raft_consensus.cc:357] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.590194 1166 raft_consensus.cc:383] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.590246 1166 raft_consensus.cc:738] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.590437 1166 consensus_queue.cc:260] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.591068 1166 ts_tablet_manager.cc:1428] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:06.591161 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c431abffd05c4136b7510c60eee2a8b1
I20250625 13:52:06.591352 1166 tablet_bootstrap.cc:492] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.595227 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c431abffd05c4136b7510c60eee2a8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.595584 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 463) took 6347 ms. Trace:
I20250625 13:52:06.595656 1168 rpcz_store.cc:276] 0625 13:52:00.248083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.333085 (+ 85002us) service_pool.cc:225] Handling call
0625 13:52:06.595467 (+6262382us) ts_tablet_manager.cc:1145] Deleting tablet cbf7e30cdff245208dacc6963fe88c7d
0625 13:52:06.595476 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.595555 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6262201}
I20250625 13:52:06.596210 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 464) took 6288 ms. Trace:
I20250625 13:52:06.596283 1168 rpcz_store.cc:276] 0625 13:52:00.307421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.333279 (+ 25858us) service_pool.cc:225] Handling call
0625 13:52:06.596114 (+6262835us) ts_tablet_manager.cc:1145] Deleting tablet ed19663d21d84c9e90d4f4dda055420d
0625 13:52:06.596122 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.596194 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6262704}
I20250625 13:52:06.596699 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 465) took 6206 ms. Trace:
I20250625 13:52:06.596789 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c431abffd05c4136b7510c60eee2a8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.596765 1168 rpcz_store.cc:276] 0625 13:52:00.390075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.390129 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:06.596606 (+6206477us) ts_tablet_manager.cc:1145] Deleting tablet efc9536a716f4b9f9fc4af1637b3cd80
0625 13:52:06.596614 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.596684 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6206141}
I20250625 13:52:06.597152 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 466) took 6032 ms. Trace:
I20250625 13:52:06.592942 1166 tablet_bootstrap.cc:654] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.597224 1168 rpcz_store.cc:276] 0625 13:52:00.564790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.567737 (+ 2947us) service_pool.cc:225] Handling call
0625 13:52:06.597061 (+6029324us) ts_tablet_manager.cc:1145] Deleting tablet afa1c198dd6541809843244c7cc18194
0625 13:52:06.597070 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.597138 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6029108}
I20250625 13:52:06.595595 741 tablet_service.cc:1515] Processing DeleteTablet for tablet c431abffd05c4136b7510c60eee2a8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.597617 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 467) took 5914 ms. Trace:
I20250625 13:52:06.597682 1168 rpcz_store.cc:276] 0625 13:52:00.683001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.683607 (+ 606us) service_pool.cc:225] Handling call
0625 13:52:06.597516 (+5913909us) ts_tablet_manager.cc:1145] Deleting tablet c605d5b64c164c1594b0a40cfa2c4cf6
0625 13:52:06.597524 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.597604 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5878020}
I20250625 13:52:06.598011 1168 tablet_replica.cc:331] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:06.598201 1168 raft_consensus.cc:2241] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.598347 1168 raft_consensus.cc:2270] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.598873 1168 ts_tablet_manager.cc:1905] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.599083 1710 ts_tablet_manager.cc:1905] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:06.595114 1963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:52:06.604046 1165 ts_tablet_manager.cc:1905] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:06.604172 1961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:52:06.605222 1170 tablet_bootstrap.cc:492] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.605351 1170 ts_tablet_manager.cc:1397] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.002s
I20250625 13:52:06.606061 1170 raft_consensus.cc:357] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.606259 1170 raft_consensus.cc:383] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.606314 1170 raft_consensus.cc:738] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:06.609973 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet cbf7e30cdff245208dacc6963fe88c7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbf7e30cdff245208dacc6963fe88c7d already in progress: creating tablet
W20250625 13:52:06.610431 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ed19663d21d84c9e90d4f4dda055420d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed19663d21d84c9e90d4f4dda055420d already in progress: creating tablet
W20250625 13:52:06.610678 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet efc9536a716f4b9f9fc4af1637b3cd80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efc9536a716f4b9f9fc4af1637b3cd80 already in progress: creating tablet
W20250625 13:52:06.610925 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet afa1c198dd6541809843244c7cc18194 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afa1c198dd6541809843244c7cc18194 already in progress: creating tablet
W20250625 13:52:06.611183 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c605d5b64c164c1594b0a40cfa2c4cf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c605d5b64c164c1594b0a40cfa2c4cf6 already in progress: creating tablet
I20250625 13:52:06.612167 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 653267b468ef4958b98af0ed6a7bf4d3 (table default.Table_fd444f4e61ca4ecb9dc9ee9660e8e36b [id=6bfad46432b7473f88d77bfcfa2a4e9a]). Expected version 1 got 0
I20250625 13:52:06.612692 1710 ts_tablet_manager.cc:1918] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.615195 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4282c7f41da742dca77dc5965c226520" } new_table_name: "default.Table_353eb61ad57d4e8084b18257adf1f9a1" modify_external_catalogs: true
I20250625 13:52:06.615550 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.617854 1710 log.cc:1199] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/8b7062c34fd143bf86b743055a27db8c
I20250625 13:52:06.618182 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 68a7712455694a1cbead413230620827
I20250625 13:52:06.618321 1710 ts_tablet_manager.cc:1939] T 8b7062c34fd143bf86b743055a27db8c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.618721 590 catalog_manager.cc:6652] ReplaceTablet: tablet 68a7712455694a1cbead413230620827 deleted and replaced by tablet c27caba6c4d14d51b10bf04f131c6a3c
I20250625 13:52:06.619129 1170 consensus_queue.cc:260] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.619266 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c366da4fb138453386d97fba0f5c91be
I20250625 13:52:06.619652 1170 ts_tablet_manager.cc:1428] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250625 13:52:06.619959 1170 tablet_bootstrap.cc:492] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.620136 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 74884bf313724d678d3b5f5869f09f97
I20250625 13:52:06.621318 1170 tablet_bootstrap.cc:654] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.623210 1165 ts_tablet_manager.cc:1918] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.623368 1165 log.cc:1199] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/0b129117db064a15b27e82439637cf22
I20250625 13:52:06.623705 1165 ts_tablet_manager.cc:1939] T 0b129117db064a15b27e82439637cf22 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:06.624809 1165 tablet_replica.cc:331] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:06.625065 1165 raft_consensus.cc:2241] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.625180 1165 raft_consensus.cc:2270] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.626499 1166 tablet_bootstrap.cc:492] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.626609 1166 ts_tablet_manager.cc:1397] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.003s
I20250625 13:52:06.626724 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 68a7712455694a1cbead413230620827 with delete_type TABLET_DATA_DELETED (replaced by tablet c27caba6c4d14d51b10bf04f131c6a3c) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.626950 757 tablet_service.cc:1515] Processing DeleteTablet for tablet c366da4fb138453386d97fba0f5c91be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.627292 1166 raft_consensus.cc:357] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.627460 1166 raft_consensus.cc:383] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.627501 1166 raft_consensus.cc:738] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.627887 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 0b129117db064a15b27e82439637cf22 (table default.Table_78be58e64e564e6ab48ccee86fbe5279 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]) successfully deleted
I20250625 13:52:06.628330 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 74884bf313724d678d3b5f5869f09f97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.629455 1794 server_base.cc:1048] running on GCE node
I20250625 13:52:06.629587 1165 ts_tablet_manager.cc:1905] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:06.630091 1966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:52:06.630779 1168 ts_tablet_manager.cc:1918] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.630889 1168 log.cc:1199] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/799f2edd960d45c1ace9195283a042e2
I20250625 13:52:06.631147 1168 ts_tablet_manager.cc:1939] T 799f2edd960d45c1ace9195283a042e2 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:06.631122 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 8b7062c34fd143bf86b743055a27db8c (table default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]) successfully deleted
I20250625 13:52:06.631244 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250625 13:52:06.631385 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c431abffd05c4136b7510c60eee2a8b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c431abffd05c4136b7510c60eee2a8b1 already in progress: creating tablet
I20250625 13:52:06.631621 1166 consensus_queue.cc:260] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.631906 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 468) took 5922 ms. Trace:
I20250625 13:52:06.631982 1168 rpcz_store.cc:276] 0625 13:52:00.709299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:00.709459 (+ 160us) service_pool.cc:225] Handling call
0625 13:52:06.597969 (+5888510us) ts_tablet_manager.cc:1145] Deleting tablet 799f2edd960d45c1ace9195283a042e2
0625 13:52:06.597978 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.602111 (+ 4133us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.630679 (+ 28568us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.631881 (+ 1202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5877943}
I20250625 13:52:06.632287 1168 tablet_replica.cc:331] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:06.632448 1168 raft_consensus.cc:2241] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:06.632551 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 68a7712455694a1cbead413230620827 with delete_type TABLET_DATA_DELETED (replaced by tablet c27caba6c4d14d51b10bf04f131c6a3c) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.632751 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet c366da4fb138453386d97fba0f5c91be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.632761 1168 raft_consensus.cc:2270] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.632894 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 74884bf313724d678d3b5f5869f09f97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.633252 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 68a7712455694a1cbead413230620827 with delete_type TABLET_DATA_DELETED (replaced by tablet c27caba6c4d14d51b10bf04f131c6a3c) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.633457 889 tablet_service.cc:1515] Processing DeleteTablet for tablet c366da4fb138453386d97fba0f5c91be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.633632 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 74884bf313724d678d3b5f5869f09f97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.634523 1170 tablet_bootstrap.cc:492] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.634644 1170 ts_tablet_manager.cc:1397] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20250625 13:52:06.635107 1166 ts_tablet_manager.cc:1428] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250625 13:52:06.635285 1170 raft_consensus.cc:357] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.635462 1166 tablet_bootstrap.cc:492] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.635460 1170 raft_consensus.cc:383] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.636193 1170 raft_consensus.cc:738] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.636368 1171 tablet_bootstrap.cc:492] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:06.636482 1171 ts_tablet_manager.cc:1397] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.004s
I20250625 13:52:06.636405 1170 consensus_queue.cc:260] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.636829 1170 ts_tablet_manager.cc:1428] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:06.637063 1166 tablet_bootstrap.cc:654] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.637096 1170 tablet_bootstrap.cc:492] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.637319 1171 raft_consensus.cc:357] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.637807 1171 raft_consensus.cc:383] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.637871 1171 raft_consensus.cc:738] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.638092 1171 consensus_queue.cc:260] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.638499 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3e4c0ca74c2f4252a4dc3c7e85afc3d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.639112 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e4c0ca74c2f4252a4dc3c7e85afc3d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.639286 1170 tablet_bootstrap.cc:654] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.641889 1171 ts_tablet_manager.cc:1428] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:06.642113 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:52:06.642203 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4187a17a1e4245cf8d7d644bb7713836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.642824 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4187a17a1e4245cf8d7d644bb7713836 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.643508 1794 hybrid_clock.cc:648] HybridClock initialized: now 1750859526643420 us; error 76 us; skew 500 ppm
I20250625 13:52:06.643998 1794 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:52:06.644233 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_08d0c030c2cd4f46ad44c12bfee6842d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.644410 1165 ts_tablet_manager.cc:1918] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.644526 1165 log.cc:1199] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/be3313ec5dfb41b68662d3ff4c8c94f9
W20250625 13:52:06.644814 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08d0c030c2cd4f46ad44c12bfee6842d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.644918 1165 ts_tablet_manager.cc:1939] T be3313ec5dfb41b68662d3ff4c8c94f9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:06.646209 1170 tablet_bootstrap.cc:492] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.646350 1170 ts_tablet_manager.cc:1397] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20250625 13:52:06.647167 1165 tablet_replica.cc:331] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:06.647081 1170 raft_consensus.cc:357] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.647250 1170 raft_consensus.cc:383] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.647301 1170 raft_consensus.cc:738] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.647367 1165 raft_consensus.cc:2241] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.647225 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet be3313ec5dfb41b68662d3ff4c8c94f9 (table default.Table_fb311c42df024421b325596e0cb7a778 [id=7b67603755e7428496524ee716b435e9]) successfully deleted
I20250625 13:52:06.647531 1165 raft_consensus.cc:2270] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.647543 1170 consensus_queue.cc:260] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:06.647784 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c431abffd05c4136b7510c60eee2a8b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c431abffd05c4136b7510c60eee2a8b1 already in progress: creating tablet
I20250625 13:52:06.648010 1170 ts_tablet_manager.cc:1428] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:06.648648 1170 tablet_bootstrap.cc:492] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.649879 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 799f2edd960d45c1ace9195283a042e2 (table default.Table_9b2f5c3f9f4d4ea69644097de7596642 [id=82115aa29a8f456b8d7c2f3d9caa17e1]) successfully deleted
I20250625 13:52:06.651392 1170 tablet_bootstrap.cc:654] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.651840 1171 tablet_bootstrap.cc:492] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:06.652797 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 68a7712455694a1cbead413230620827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68a7712455694a1cbead413230620827 already in progress: creating tablet
I20250625 13:52:06.655989 1171 tablet_bootstrap.cc:654] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.660023 1170 tablet_bootstrap.cc:492] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.660143 1170 ts_tablet_manager.cc:1397] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
W20250625 13:52:06.661106 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c366da4fb138453386d97fba0f5c91be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c366da4fb138453386d97fba0f5c91be already in progress: creating tablet
I20250625 13:52:06.661772 1170 raft_consensus.cc:357] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.661954 1170 raft_consensus.cc:383] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.662003 1170 raft_consensus.cc:738] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.662191 1170 consensus_queue.cc:260] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.662683 1170 ts_tablet_manager.cc:1428] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250625 13:52:06.665771 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 74884bf313724d678d3b5f5869f09f97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74884bf313724d678d3b5f5869f09f97 already in progress: creating tablet
I20250625 13:52:06.665874 1170 tablet_bootstrap.cc:492] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.667714 1170 tablet_bootstrap.cc:654] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.668974 1165 ts_tablet_manager.cc:1905] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.672526 1794 webserver.cc:469] Webserver started at http://127.0.104.62:44303/ using document root <none> and password file <none>
I20250625 13:52:06.673686 1794 fs_manager.cc:362] Metadata directory not provided
I20250625 13:52:06.673820 1794 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:52:06.677034 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6e8b9b732f8948d2b4c1a81a1de4489a
I20250625 13:52:06.677238 1165 ts_tablet_manager.cc:1918] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.677573 592 catalog_manager.cc:6652] ReplaceTablet: tablet 6e8b9b732f8948d2b4c1a81a1de4489a deleted and replaced by tablet 93a0b94879874446b569b3cb3250b185
I20250625 13:52:06.677753 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e8b9b732f8948d2b4c1a81a1de4489a with delete_type TABLET_DATA_DELETED (replaced by tablet 93a0b94879874446b569b3cb3250b185) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.677917 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e8b9b732f8948d2b4c1a81a1de4489a with delete_type TABLET_DATA_DELETED (replaced by tablet 93a0b94879874446b569b3cb3250b185) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:06.678306 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d139478986e346e2ac833ce353cff7e1 consensus state has no leader
I20250625 13:52:06.678562 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e8b9b732f8948d2b4c1a81a1de4489a with delete_type TABLET_DATA_DELETED (replaced by tablet 93a0b94879874446b569b3cb3250b185) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:06.679392 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6e8b9b732f8948d2b4c1a81a1de4489a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e8b9b732f8948d2b4c1a81a1de4489a already in progress: creating tablet
I20250625 13:52:06.680414 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cbe4046742b34f1ab0b1610b5d6ee01e
I20250625 13:52:06.680851 1170 tablet_bootstrap.cc:492] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.680975 1170 ts_tablet_manager.cc:1397] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250625 13:52:06.681306 594 catalog_manager.cc:6652] ReplaceTablet: tablet cbe4046742b34f1ab0b1610b5d6ee01e deleted and replaced by tablet 066db01f791542a38cfb6e408c4836ed
I20250625 13:52:06.680585 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5d478ac74f4f46c6a29a6ac031152498
I20250625 13:52:06.681728 1170 raft_consensus.cc:357] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.681908 1170 raft_consensus.cc:383] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.681960 1170 raft_consensus.cc:738] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.682147 1170 consensus_queue.cc:260] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.682556 1170 ts_tablet_manager.cc:1428] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:06.682893 1170 tablet_bootstrap.cc:492] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.683537 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e with delete_type TABLET_DATA_DELETED (replaced by tablet 066db01f791542a38cfb6e408c4836ed) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.684127 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d478ac74f4f46c6a29a6ac031152498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.684161 881 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e with delete_type TABLET_DATA_DELETED (replaced by tablet 066db01f791542a38cfb6e408c4836ed) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.684490 757 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e with delete_type TABLET_DATA_DELETED (replaced by tablet 066db01f791542a38cfb6e408c4836ed) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.684901 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d478ac74f4f46c6a29a6ac031152498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.685389 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d478ac74f4f46c6a29a6ac031152498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.686787 1170 tablet_bootstrap.cc:654] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:06.687953 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet cbe4046742b34f1ab0b1610b5d6ee01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe4046742b34f1ab0b1610b5d6ee01e already in progress: creating tablet
W20250625 13:52:06.688637 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5d478ac74f4f46c6a29a6ac031152498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d478ac74f4f46c6a29a6ac031152498 already in progress: creating tablet
I20250625 13:52:06.689210 1165 log.cc:1199] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/9c1deb52366c404cbdc08076d4bcb3d1
I20250625 13:52:06.689716 1165 ts_tablet_manager.cc:1939] T 9c1deb52366c404cbdc08076d4bcb3d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:06.690999 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 9c1deb52366c404cbdc08076d4bcb3d1 (table default.Table_b2b0087eeea54823809b847648141d81 [id=253f3f5c5b8d4051800dc823b4ebaec9]) successfully deleted
I20250625 13:52:06.691565 1170 tablet_bootstrap.cc:492] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.691689 1170 ts_tablet_manager.cc:1397] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
W20250625 13:52:06.691702 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 68a7712455694a1cbead413230620827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68a7712455694a1cbead413230620827 already in progress: creating tablet
W20250625 13:52:06.692327 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c366da4fb138453386d97fba0f5c91be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c366da4fb138453386d97fba0f5c91be already in progress: creating tablet
I20250625 13:52:06.692705 1170 raft_consensus.cc:357] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:06.692977 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 74884bf313724d678d3b5f5869f09f97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74884bf313724d678d3b5f5869f09f97 already in progress: creating tablet
I20250625 13:52:06.693027 1170 raft_consensus.cc:383] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.693086 1170 raft_consensus.cc:738] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.693194 1794 fs_manager.cc:714] Time spent opening directory manager: real 0.016s user 0.004s sys 0.002s
I20250625 13:52:06.693320 1170 consensus_queue.cc:260] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.693782 1170 ts_tablet_manager.cc:1428] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 13:52:06.693948 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6e8b9b732f8948d2b4c1a81a1de4489a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e8b9b732f8948d2b4c1a81a1de4489a already in progress: creating tablet
I20250625 13:52:06.694413 1170 tablet_bootstrap.cc:492] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:06.694792 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5d478ac74f4f46c6a29a6ac031152498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d478ac74f4f46c6a29a6ac031152498 already in progress: creating tablet
W20250625 13:52:06.695416 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet cbe4046742b34f1ab0b1610b5d6ee01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe4046742b34f1ab0b1610b5d6ee01e already in progress: creating tablet
I20250625 13:52:06.697363 1170 tablet_bootstrap.cc:654] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.697856 1985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:52:06.700613 1794 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250625 13:52:06.701102 1794 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
uuid: "3fec9d8a79054a5991ae90d8a3101e32"
format_stamp: "Formatted at 2025-06-25 13:51:48 on dist-test-slave-x7jc"
I20250625 13:52:06.702530 1794 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:52:06.705860 1748 raft_consensus.cc:491] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.706077 1748 raft_consensus.cc:513] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.706935 1748 leader_election.cc:290] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.708738 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f065b4e3218b4e6e8a2ac3a70eede13b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.709565 851 leader_election.cc:343] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f065b4e3218b4e6e8a2ac3a70eede13b
W20250625 13:52:06.710125 849 leader_election.cc:343] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.710247 849 leader_election.cc:304] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.710830 1748 raft_consensus.cc:2747] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.633260 1168 ts_tablet_manager.cc:1905] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:06.733023 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd036ec08f774f5393067ce20bb07610 consensus state has no leader
I20250625 13:52:06.740666 1837 raft_consensus.cc:491] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.740832 1837 raft_consensus.cc:513] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.741359 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_50ec7255c1514670a780c11526d562a8" } new_table_name: "default.Table_6895050a06634c5797ba2c2af0f4da14" modify_external_catalogs: true
I20250625 13:52:06.741469 1837 leader_election.cc:290] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.742056 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_45ffe0341a354703ab5ab6c66bd27f18" } new_table_name: "default.Table_e6b348f960d5446ca1bfe847c5142516" modify_external_catalogs: true
I20250625 13:52:06.742916 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "653267b468ef4958b98af0ed6a7bf4d3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:06.743131 774 raft_consensus.cc:2466] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:06.743367 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_31c75178c8ab41378e5a92bb7b2ccb9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.743921 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31c75178c8ab41378e5a92bb7b2ccb9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.744040 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fa6f50b15eff4238819930c9bf68b192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.744091 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "653267b468ef4958b98af0ed6a7bf4d3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.744588 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa6f50b15eff4238819930c9bf68b192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.745127 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_80f1e7ae1dce4246b10319e14d4dcaa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.745568 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80f1e7ae1dce4246b10319e14d4dcaa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.746711 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fd444f4e61ca4ecb9dc9ee9660e8e36b" } new_table_name: "default.Table_107a64d05b814c5a9cc7b956f38af315" modify_external_catalogs: true
I20250625 13:52:06.747887 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.747864 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_64adeba774054ea5a73be69a92bf079e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.748026 851 leader_election.cc:304] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:06.748183 1710 tablet_replica.cc:331] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:06.748302 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64adeba774054ea5a73be69a92bf079e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.748395 1710 raft_consensus.cc:2241] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.748538 1842 raft_consensus.cc:2802] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:06.748622 1842 raft_consensus.cc:491] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:06.748697 1842 raft_consensus.cc:3058] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:06.748739 1710 raft_consensus.cc:2270] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.749699 1710 ts_tablet_manager.cc:1905] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:06.753726 849 leader_election.cc:343] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.757819 1510 raft_consensus.cc:491] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.757943 1510 raft_consensus.cc:513] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.758369 1710 ts_tablet_manager.cc:1918] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.758487 1710 log.cc:1199] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/8e29e7b55570483b80c6e7a1f227fa7b
I20250625 13:52:06.758504 1510 leader_election.cc:290] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.758841 1710 ts_tablet_manager.cc:1939] T 8e29e7b55570483b80c6e7a1f227fa7b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.760054 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd39216b01054ff38ec9175d7a2f87f2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:06.760109 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 8e29e7b55570483b80c6e7a1f227fa7b (table default.Table_2b3cf346c6724405a0e96eef702fc43e [id=4891101f0201404ab9d5c88c094ce3ff]) successfully deleted
W20250625 13:52:06.760664 851 leader_election.cc:343] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: fd39216b01054ff38ec9175d7a2f87f2
W20250625 13:52:06.761051 849 leader_election.cc:343] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.761165 849 leader_election.cc:304] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.761585 1837 raft_consensus.cc:2747] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.764868 1166 tablet_bootstrap.cc:492] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.765095 1166 ts_tablet_manager.cc:1397] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.130s user 0.039s sys 0.038s
I20250625 13:52:06.751233 1842 raft_consensus.cc:513] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.766036 1166 raft_consensus.cc:357] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.766209 1166 raft_consensus.cc:383] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.766273 1166 raft_consensus.cc:738] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.766538 1166 consensus_queue.cc:260] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.767381 1166 ts_tablet_manager.cc:1428] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:06.767853 1166 tablet_bootstrap.cc:492] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.768358 1842 leader_election.cc:290] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.769102 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "653267b468ef4958b98af0ed6a7bf4d3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:06.769529 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "653267b468ef4958b98af0ed6a7bf4d3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:06.769677 774 raft_consensus.cc:3058] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:06.770246 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e29e7b55570483b80c6e7a1f227fa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.770318 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 637847cac1724bfeb629c4d4013396ca with delete_type TABLET_DATA_DELETED (replaced by tablet b14d00c93a964edd815122e3723f89e7) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.770444 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 58aa3c21191e4bea9c5395d6e32b8717 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.770974 1710 tablet_replica.cc:331] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.771193 1710 raft_consensus.cc:2241] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.771385 1710 raft_consensus.cc:2270] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:06.771588 849 leader_election.cc:343] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.773089 774 raft_consensus.cc:2466] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:06.773686 851 leader_election.cc:304] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:06.774008 1842 raft_consensus.cc:2802] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:06.774160 1842 raft_consensus.cc:695] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:06.774466 1842 consensus_queue.cc:237] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.774796 1710 ts_tablet_manager.cc:1905] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.775918 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet af05fba6fe2042aab5dd47b30e4dfcf3 with delete_type TABLET_DATA_DELETED (replaced by tablet abc44a41b30e4a53977ec81d6905612f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.776098 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a22ade8cefac4140b08882c311ea5181 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.776207 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c666ff2d6b9b4ed4a18e370826c143fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.776710 1166 tablet_bootstrap.cc:654] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:06.777365 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8e29e7b55570483b80c6e7a1f227fa7b because the tablet was not found. No further retry: Not found: Tablet not found: 8e29e7b55570483b80c6e7a1f227fa7b
I20250625 13:52:06.781000 1710 ts_tablet_manager.cc:1918] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.781116 1710 log.cc:1199] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/637847cac1724bfeb629c4d4013396ca
I20250625 13:52:06.781416 1710 ts_tablet_manager.cc:1939] T 637847cac1724bfeb629c4d4013396ca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.781967 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 637847cac1724bfeb629c4d4013396ca (table default.Table_295dd4f121234e659090b8e4cf70faee [id=47cd45886fc94cd7ba5cafa60838d601]) successfully deleted
I20250625 13:52:06.782126 1710 tablet_replica.cc:331] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.782182 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_585826da245840dab264effeab34e150" } modify_external_catalogs: true
I20250625 13:52:06.782285 1710 raft_consensus.cc:2241] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.782356 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_585826da245840dab264effeab34e150" } modify_external_catalogs: true
I20250625 13:52:06.782430 1710 raft_consensus.cc:2270] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.783494 1166 tablet_bootstrap.cc:492] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.783628 1166 ts_tablet_manager.cc:1397] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250625 13:52:06.784415 1166 raft_consensus.cc:357] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.784687 1166 raft_consensus.cc:383] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.784762 1166 raft_consensus.cc:738] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.785061 1166 consensus_queue.cc:260] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.787146 1166 ts_tablet_manager.cc:1428] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:06.787755 1166 tablet_bootstrap.cc:492] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.790829 1166 tablet_bootstrap.cc:654] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.791251 1776 raft_consensus.cc:491] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.791389 1776 raft_consensus.cc:513] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.791637 1710 ts_tablet_manager.cc:1905] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.791946 1776 leader_election.cc:290] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:06.792558 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.796157 711 leader_election.cc:343] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.799400 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:06.800017 711 leader_election.cc:343] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.800127 711 leader_election.cc:304] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:06.800486 1776 raft_consensus.cc:2747] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.812419 1171 tablet_bootstrap.cc:492] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:06.812635 1171 ts_tablet_manager.cc:1397] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.161s user 0.001s sys 0.004s
I20250625 13:52:06.813632 1171 raft_consensus.cc:357] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.813933 1171 raft_consensus.cc:383] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.814050 1171 raft_consensus.cc:738] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.814393 1171 consensus_queue.cc:260] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.815210 1171 ts_tablet_manager.cc:1428] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:06.815595 1171 tablet_bootstrap.cc:492] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:06.817610 1171 tablet_bootstrap.cc:654] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.821583 1710 ts_tablet_manager.cc:1918] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.821710 1710 log.cc:1199] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/58aa3c21191e4bea9c5395d6e32b8717
I20250625 13:52:06.821990 1710 ts_tablet_manager.cc:1939] T 58aa3c21191e4bea9c5395d6e32b8717 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.822373 1710 tablet_replica.cc:331] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.822532 1710 raft_consensus.cc:2241] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.822554 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 58aa3c21191e4bea9c5395d6e32b8717 (table default.Table_759e39cb7c154d6281c46789f97f0691 [id=5efdc94a9cf74a3daa545d4d93684eaa]) successfully deleted
I20250625 13:52:06.822657 1710 raft_consensus.cc:2270] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.826982 1710 ts_tablet_manager.cc:1905] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:06.831142 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf7463962f39468c85fb3dc2edcf608f consensus state has no leader
I20250625 13:52:06.831283 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9b61484dc29b44ac9d0afd92e1da0ef3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:06.831564 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9d8551fdf4e043cd90e2ce8228c3db7c" } modify_external_catalogs: true
I20250625 13:52:06.831717 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9d8551fdf4e043cd90e2ce8228c3db7c" } modify_external_catalogs: true
W20250625 13:52:06.831998 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b61484dc29b44ac9d0afd92e1da0ef3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.832520 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_003becbbf12b448b8d9581a65994b657" } modify_external_catalogs: true
I20250625 13:52:06.832672 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_003becbbf12b448b8d9581a65994b657" } modify_external_catalogs: true
I20250625 13:52:06.833506 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40856c021b18443fb5feae09a82e09ad" } modify_external_catalogs: true
I20250625 13:52:06.833694 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40856c021b18443fb5feae09a82e09ad" } modify_external_catalogs: true
W20250625 13:52:06.834138 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f3de20ac5f945bdb62535258880186f consensus state has no leader
I20250625 13:52:06.834652 1166 tablet_bootstrap.cc:492] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.834805 1166 ts_tablet_manager.cc:1397] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.018s
I20250625 13:52:06.835767 1166 raft_consensus.cc:357] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.835994 1166 raft_consensus.cc:383] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.836093 1166 raft_consensus.cc:738] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.853938 1166 consensus_queue.cc:260] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.854077 1168 ts_tablet_manager.cc:1918] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:06.854212 1168 log.cc:1199] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/93f449c80b604a6498f6fd1d1ba4770a
I20250625 13:52:06.854607 1168 ts_tablet_manager.cc:1939] T 93f449c80b604a6498f6fd1d1ba4770a P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:06.854830 1166 ts_tablet_manager.cc:1428] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s
I20250625 13:52:06.855165 1166 tablet_bootstrap.cc:492] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.855449 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 488) took 5373 ms. Trace:
I20250625 13:52:06.855546 1168 rpcz_store.cc:276] 0625 13:52:01.481959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.482313 (+ 354us) service_pool.cc:225] Handling call
0625 13:52:06.632242 (+5149929us) ts_tablet_manager.cc:1145] Deleting tablet 93f449c80b604a6498f6fd1d1ba4770a
0625 13:52:06.632251 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.800555 (+168304us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.853966 (+ 53411us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:06.855425 (+ 1459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5149716}
I20250625 13:52:06.855990 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 93f449c80b604a6498f6fd1d1ba4770a (table default.Table_7ccc261a6367449583a5c30495bfe2a9 [id=5436e1b638d14b00a66842a43cc5c085]) successfully deleted
W20250625 13:52:06.856739 849 consensus_peers.cc:487] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:06.857299 1166 tablet_bootstrap.cc:654] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.861255 1837 raft_consensus.cc:491] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.861428 1837 raft_consensus.cc:513] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.862047 1837 leader_election.cc:290] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:06.868389 851 leader_election.cc:343] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ecb9a98e402b4da8a9c2bd8317cef532
I20250625 13:52:06.868899 1166 tablet_bootstrap.cc:492] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:06.869078 1166 ts_tablet_manager.cc:1397] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20250625 13:52:06.869306 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecb9a98e402b4da8a9c2bd8317cef532" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:06.869912 1166 raft_consensus.cc:357] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.870065 1710 ts_tablet_manager.cc:1918] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:06.870080 849 leader_election.cc:343] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.870201 1710 log.cc:1199] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/af05fba6fe2042aab5dd47b30e4dfcf3
I20250625 13:52:06.870199 849 leader_election.cc:304] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.870231 1166 raft_consensus.cc:383] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.870334 1166 raft_consensus.cc:738] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.870496 1710 ts_tablet_manager.cc:1939] T af05fba6fe2042aab5dd47b30e4dfcf3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.870551 1166 consensus_queue.cc:260] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.870785 1842 raft_consensus.cc:2747] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.871100 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet af05fba6fe2042aab5dd47b30e4dfcf3 (table default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]) successfully deleted
I20250625 13:52:06.872431 1170 tablet_bootstrap.cc:492] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.872664 1170 ts_tablet_manager.cc:1397] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.179s user 0.024s sys 0.026s
I20250625 13:52:06.873447 1170 raft_consensus.cc:357] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.873621 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 490) took 5307 ms. Trace:
I20250625 13:52:06.873669 1170 raft_consensus.cc:383] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.873710 1168 rpcz_store.cc:276] 0625 13:52:01.566397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.566461 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:06.873488 (+5307027us) ts_tablet_manager.cc:1145] Deleting tablet 063a793e0d7747668d2728fdc2160b5e
0625 13:52:06.873501 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:06.873604 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5306786}
I20250625 13:52:06.873821 1170 raft_consensus.cc:738] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:06.874164 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 063a793e0d7747668d2728fdc2160b5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 063a793e0d7747668d2728fdc2160b5e already in progress: creating tablet
I20250625 13:52:06.872529 1166 ts_tablet_manager.cc:1428] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:06.874169 1170 consensus_queue.cc:260] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.875073 1170 ts_tablet_manager.cc:1428] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:06.876107 1170 tablet_bootstrap.cc:492] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.876148 1166 tablet_bootstrap.cc:492] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:06.878717 1166 tablet_bootstrap.cc:654] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.880610 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c0a3ea18bab944749d45da9b3139f8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.883854 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0998ad87fcc145999f6430e0de7b1304
I20250625 13:52:06.884235 1170 tablet_bootstrap.cc:654] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.885025 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 0998ad87fcc145999f6430e0de7b1304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.885756 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 0998ad87fcc145999f6430e0de7b1304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.886746 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 0998ad87fcc145999f6430e0de7b1304 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.886976 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3bf17a9252fa4b4d9150da868f39d897" } new_table_name: "default.Table_cc87fc1ee177485192b77fe418e55945" modify_external_catalogs: true
I20250625 13:52:06.887033 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6a4abf185c2b44d3ba20510778eead0a" } modify_external_catalogs: true
I20250625 13:52:06.887228 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6a4abf185c2b44d3ba20510778eead0a" } modify_external_catalogs: true
W20250625 13:52:06.887686 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0998ad87fcc145999f6430e0de7b1304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0998ad87fcc145999f6430e0de7b1304 already in progress: creating tablet
I20250625 13:52:06.891827 1710 tablet_replica.cc:331] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.892045 1710 raft_consensus.cc:2241] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.892227 1710 raft_consensus.cc:2270] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.892982 1710 ts_tablet_manager.cc:1905] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.895853 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 474356bf56b94681a8dc9071c0d8e79a
I20250625 13:52:06.896775 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 474356bf56b94681a8dc9071c0d8e79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.897040 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 474356bf56b94681a8dc9071c0d8e79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.897099 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 474356bf56b94681a8dc9071c0d8e79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.900568 1170 tablet_bootstrap.cc:492] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.900701 1170 ts_tablet_manager.cc:1397] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
W20250625 13:52:06.900689 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 474356bf56b94681a8dc9071c0d8e79a because the tablet was not found. No further retry: Not found: Tablet not found: 474356bf56b94681a8dc9071c0d8e79a
I20250625 13:52:06.901407 1170 raft_consensus.cc:357] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.901572 1170 raft_consensus.cc:383] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.901623 1170 raft_consensus.cc:738] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.903856 1171 tablet_bootstrap.cc:492] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:06.904086 1171 ts_tablet_manager.cc:1397] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.002s
I20250625 13:52:06.905046 1171 raft_consensus.cc:357] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.905467 1171 raft_consensus.cc:383] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.905596 1171 raft_consensus.cc:738] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.905959 1171 consensus_queue.cc:260] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.907429 1170 consensus_queue.cc:260] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.907930 1170 ts_tablet_manager.cc:1428] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:06.908300 1170 tablet_bootstrap.cc:492] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.909335 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 535812d46d6444fda85d9767c6ca7e62 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:06.910379 1170 tablet_bootstrap.cc:654] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.917129 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_d6bab22e154a444f90a895591fa09ca3 [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 3 got 0
I20250625 13:52:06.917498 1710 ts_tablet_manager.cc:1918] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.917598 1710 log.cc:1199] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a22ade8cefac4140b08882c311ea5181
I20250625 13:52:06.917904 1710 ts_tablet_manager.cc:1939] T a22ade8cefac4140b08882c311ea5181 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.918465 1710 tablet_replica.cc:331] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.918648 1710 raft_consensus.cc:2241] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:06.918743 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b493f32299ba42159c594849e0fe39c4 has no consensus state
I20250625 13:52:06.918788 1710 raft_consensus.cc:2270] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.918861 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a22ade8cefac4140b08882c311ea5181 (table default.Table_9dfb8c48d5144c808567a16a30cc199e [id=f55ecdcac2234415a085be00cacad571]) successfully deleted
I20250625 13:52:06.924258 1710 ts_tablet_manager.cc:1905] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.925242 1842 consensus_queue.cc:1035] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:06.926021 1170 tablet_bootstrap.cc:492] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.926141 1170 ts_tablet_manager.cc:1397] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20250625 13:52:06.926807 1170 raft_consensus.cc:357] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.926962 1170 raft_consensus.cc:383] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.927016 1170 raft_consensus.cc:738] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.927217 1170 consensus_queue.cc:260] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.928645 1170 ts_tablet_manager.cc:1428] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:06.928915 1170 tablet_bootstrap.cc:492] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:06.930998 1170 tablet_bootstrap.cc:654] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.935258 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2733881e50814fb49b1cb5ee2e42f772" } new_table_name: "default.Table_87a6dacca293482bb45ecc57e7cd6711" modify_external_catalogs: true
I20250625 13:52:06.935647 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0d7502063c8f4f06bcb3f4bd4c9e8cd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.936401 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d7502063c8f4f06bcb3f4bd4c9e8cd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.937918 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_89f74db6137d499bbb022bd99ebc4243"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.938522 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89f74db6137d499bbb022bd99ebc4243 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.939991 591 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_d6bab22e154a444f90a895591fa09ca3 [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 3 got 0
I20250625 13:52:06.940941 591 master_service.cc:948] ReplaceTablet: received request to replace tablet bdbf586b82194a24a2259cbeb62d615c from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:06.947472 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 155737a8d5604eb69524c4d6bea61550
I20250625 13:52:06.948812 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 155737a8d5604eb69524c4d6bea61550 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.948890 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 155737a8d5604eb69524c4d6bea61550 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.949164 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 155737a8d5604eb69524c4d6bea61550 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:06.949640 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 155737a8d5604eb69524c4d6bea61550 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 155737a8d5604eb69524c4d6bea61550 already in progress: creating tablet
I20250625 13:52:06.952396 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1402caa3f71f4bf7b40629b140616eeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.952919 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1402caa3f71f4bf7b40629b140616eeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.954578 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet dbd2120b4b084691b6604305773e74f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.955613 1170 tablet_bootstrap.cc:492] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:06.955727 1170 ts_tablet_manager.cc:1397] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250625 13:52:06.956327 1170 raft_consensus.cc:357] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.956486 1170 raft_consensus.cc:383] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:06.956537 1170 raft_consensus.cc:738] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:06.956737 1170 consensus_queue.cc:260] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:06.959798 1170 ts_tablet_manager.cc:1428] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:06.960162 1170 tablet_bootstrap.cc:492] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:06.960990 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c6ec3a7578e4892b6193044c58b142e consensus state has no leader
I20250625 13:52:06.962725 1842 raft_consensus.cc:491] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.962886 1842 raft_consensus.cc:513] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.963539 1842 leader_election.cc:290] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.964239 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cd5303e674246ada7d36574b535b333" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.964851 849 leader_election.cc:343] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:06.964949 1710 ts_tablet_manager.cc:1918] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:06.965070 1710 log.cc:1199] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/c666ff2d6b9b4ed4a18e370826c143fc
W20250625 13:52:06.965240 851 leader_election.cc:343] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2cd5303e674246ada7d36574b535b333
I20250625 13:52:06.965343 851 leader_election.cc:304] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.965430 1710 ts_tablet_manager.cc:1939] T c666ff2d6b9b4ed4a18e370826c143fc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:06.965554 1510 raft_consensus.cc:2747] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.965978 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet c666ff2d6b9b4ed4a18e370826c143fc (table default.Table_badc8f5a2d5040af96b9e7c9758c6417 [id=8a37f6c60b414741a615d1fa7f9a4f3c]) successfully deleted
I20250625 13:52:06.968119 1171 ts_tablet_manager.cc:1428] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.064s user 0.002s sys 0.000s
I20250625 13:52:06.968420 1510 raft_consensus.cc:491] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:06.968544 1510 raft_consensus.cc:513] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:06.968736 1710 tablet_replica.cc:331] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:06.968897 1710 raft_consensus.cc:2241] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.969024 1710 raft_consensus.cc:2270] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.969082 1510 leader_election.cc:290] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:06.969470 1171 tablet_bootstrap.cc:492] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:06.969416 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bbd509e3dfb45d98314c6afe37677f6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:06.970014 849 leader_election.cc:343] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:06.970129 851 leader_election.cc:343] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2bbd509e3dfb45d98314c6afe37677f6
I20250625 13:52:06.970257 851 leader_election.cc:304] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:06.970487 1510 raft_consensus.cc:2747] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:06.971495 1171 tablet_bootstrap.cc:654] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.971905 1710 ts_tablet_manager.cc:1905] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:06.973605 590 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:06.975059 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5a6a094978314b53847a823ba43ac189"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:06.976838 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a6a094978314b53847a823ba43ac189 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:06.988158 1170 tablet_bootstrap.cc:654] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:06.989372 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 653267b468ef4958b98af0ed6a7bf4d3 (table default.Table_fd444f4e61ca4ecb9dc9ee9660e8e36b [id=6bfad46432b7473f88d77bfcfa2a4e9a]). Expected version 1 got 0
I20250625 13:52:06.989581 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 5cf73b4ab99c466abcfedf1b200e743b (table default.Table_47598e6b1611413784a8c9ab9d1c0a67 [id=8c0cf5af3aee43e9b7676c1be4842093]). Expected version 2 got 0
W20250625 13:52:06.989861 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 653267b468ef4958b98af0ed6a7bf4d3 consensus state has no leader
I20250625 13:52:06.989831 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:06.992594 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.992743 741 tablet_service.cc:1515] Processing DeleteTablet for tablet ac4d368347444846880954ced9651404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.992844 1165 tablet_replica.cc:331] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:06.991963 656 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 474356bf56b94681a8dc9071c0d8e79a
I20250625 13:52:06.993002 1165 raft_consensus.cc:2241] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:06.993134 1165 raft_consensus.cc:2270] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:06.993589 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 474356bf56b94681a8dc9071c0d8e79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:06.994230 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 474356bf56b94681a8dc9071c0d8e79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:06.994983 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:06.995623 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 474356bf56b94681a8dc9071c0d8e79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:06.995716 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 9c2317b24fd74909b133dc5a49898cf0 (DEFAULT_TABLE table=default.Table_865d72bbadc14727a2bd54342b385ee3 [id=9f53ac4763e740218e18c5e734993ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:06.996074 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2317b24fd74909b133dc5a49898cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:06.996486 1165 ts_tablet_manager.cc:1905] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.002995 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 491) took 5419 ms. Trace:
I20250625 13:52:07.003118 1168 rpcz_store.cc:276] 0625 13:52:01.583272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.583348 (+ 76us) service_pool.cc:225] Handling call
0625 13:52:07.002823 (+5419475us) ts_tablet_manager.cc:1145] Deleting tablet 4d2d6f05348e43339e7167a9cddca2ba
0625 13:52:07.002838 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.002968 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5419281}
I20250625 13:52:07.003446 1168 tablet_replica.cc:331] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:07.003662 1168 raft_consensus.cc:2241] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:07.003748 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4d2d6f05348e43339e7167a9cddca2ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2d6f05348e43339e7167a9cddca2ba already in progress: creating tablet
I20250625 13:52:07.003842 1168 raft_consensus.cc:2270] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.004289 1710 ts_tablet_manager.cc:1918] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.004374 1168 ts_tablet_manager.cc:1905] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.004611 1821 raft_consensus.cc:491] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.009596 1821 raft_consensus.cc:513] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.003051 1165 ts_tablet_manager.cc:1918] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.010411 1165 log.cc:1199] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:07.010421 1821 leader_election.cc:290] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.010746 1165 ts_tablet_manager.cc:1939] T 4fa801f7d98e4794b6ca439638f29e69 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.011325 1165 tablet_replica.cc:331] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:07.011539 1165 raft_consensus.cc:2241] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.011617 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 4fa801f7d98e4794b6ca439638f29e69 (table default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328 [id=1e5117fcc9bb41c3bb98021f499b766c]) successfully deleted
I20250625 13:52:07.011703 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:07.011754 1165 raft_consensus.cc:2270] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.011705 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8b30c229c3445b683a5015f069eca1c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:07.012679 1165 ts_tablet_manager.cc:1905] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.012930 985 leader_election.cc:304] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:07.013517 1821 raft_consensus.cc:2747] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:07.009444 1170 tablet_bootstrap.cc:492] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.013872 1170 ts_tablet_manager.cc:1397] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.001s
I20250625 13:52:07.014685 1170 raft_consensus.cc:357] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.014925 1170 raft_consensus.cc:383] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.015018 1170 raft_consensus.cc:738] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.015296 1170 consensus_queue.cc:260] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.017822 1170 ts_tablet_manager.cc:1428] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:07.018151 1170 tablet_bootstrap.cc:492] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.020133 1170 tablet_bootstrap.cc:654] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.020258 1710 log.cc:1199] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/c0a3ea18bab944749d45da9b3139f8bc
I20250625 13:52:07.020891 1710 ts_tablet_manager.cc:1939] T c0a3ea18bab944749d45da9b3139f8bc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.021456 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet c0a3ea18bab944749d45da9b3139f8bc (table default.Table_c8b06a7351fb437aa0440f6fa3898939 [id=eac16bb1f501401c9f6803136dc04116]) successfully deleted
W20250625 13:52:07.022127 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0998ad87fcc145999f6430e0de7b1304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0998ad87fcc145999f6430e0de7b1304 already in progress: creating tablet
W20250625 13:52:07.024963 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 474356bf56b94681a8dc9071c0d8e79a because the tablet was not found. No further retry: Not found: Tablet not found: 474356bf56b94681a8dc9071c0d8e79a
I20250625 13:52:07.025087 1710 tablet_replica.cc:331] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:07.025233 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 155737a8d5604eb69524c4d6bea61550 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 155737a8d5604eb69524c4d6bea61550 already in progress: creating tablet
I20250625 13:52:07.025231 1710 raft_consensus.cc:2241] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.025421 1710 raft_consensus.cc:2270] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.026326 1710 ts_tablet_manager.cc:1905] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.026597 593 catalog_manager.cc:5582] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:07.027129 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 653267b468ef4958b98af0ed6a7bf4d3 (table default.Table_107a64d05b814c5a9cc7b956f38af315 [id=6bfad46432b7473f88d77bfcfa2a4e9a]). Expected version 2 got 0
I20250625 13:52:07.028687 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_43b8ed6d4ad341448518408d6dee8613"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.029230 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43b8ed6d4ad341448518408d6dee8613 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.031087 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:07.042553 741 tablet_service.cc:1468] Processing CreateTablet for tablet 9c2317b24fd74909b133dc5a49898cf0 (DEFAULT_TABLE table=default.Table_865d72bbadc14727a2bd54342b385ee3 [id=9f53ac4763e740218e18c5e734993ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.043026 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2317b24fd74909b133dc5a49898cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.055207 1165 ts_tablet_manager.cc:1918] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.058212 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d0ce3bb2d8b04255ae3bc517960db379" } new_table_name: "default.Table_f3b3e3e07eb84bfa8a9396b3161a9114" modify_external_catalogs: true
I20250625 13:52:07.059396 1165 log.cc:1199] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ac4d368347444846880954ced9651404
I20250625 13:52:07.059741 1165 ts_tablet_manager.cc:1939] T ac4d368347444846880954ced9651404 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.060763 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ac4d368347444846880954ced9651404 (table default.Table_581fcddbe9bb428d84556e6156637b4a [id=3a7ba6e0a96c4f3fb656f9ac4f6ee82d]) successfully deleted
W20250625 13:52:07.061439 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 474356bf56b94681a8dc9071c0d8e79a because the tablet was not found. No further retry: Not found: Tablet not found: 474356bf56b94681a8dc9071c0d8e79a
I20250625 13:52:07.063372 1710 ts_tablet_manager.cc:1918] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.063616 1710 log.cc:1199] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/dbd2120b4b084691b6604305773e74f9
I20250625 13:52:07.064010 1710 ts_tablet_manager.cc:1939] T dbd2120b4b084691b6604305773e74f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.065227 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet dbd2120b4b084691b6604305773e74f9 (table default.Table_8910e07e05ad4d71bfff7d5a0d6952f4 [id=2dd7357a611c4866913dfd01fedcbe5b]) successfully deleted
I20250625 13:52:07.066231 881 tablet_service.cc:1468] Processing CreateTablet for tablet 9c2317b24fd74909b133dc5a49898cf0 (DEFAULT_TABLE table=default.Table_865d72bbadc14727a2bd54342b385ee3 [id=9f53ac4763e740218e18c5e734993ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.066659 889 tablet_service.cc:1468] Processing CreateTablet for tablet 664a83b20b344584b5c85db53bfa9021 (DEFAULT_TABLE table=default.Table_42a2e34e5d204e7188555f5e5616cc4c [id=441cf0521bab4fe08b67b987334f492f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.067512 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e1549b9e797a44f89b316a44c5e7ee2e (DEFAULT_TABLE table=default.Table_64d842d3737b421f9832c07a67be66e7 [id=39762a8b3fbd4e489f3da9b819228b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.067854 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1549b9e797a44f89b316a44c5e7ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.069986 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 35c7bf4272994e9b8a59062b1757f6f5 (DEFAULT_TABLE table=default.Table_e5270cb847db4a57979fd35c13f6fe0b [id=607b89f622c240f0917158cf08a94f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.070314 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c7bf4272994e9b8a59062b1757f6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.071538 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 81f45ab55e414eb39dec069228257976 (DEFAULT_TABLE table=default.Table_f471ebec4d67405899f155aad9d3ec83 [id=35ca80beadf742df969e352ff737586d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.071854 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f45ab55e414eb39dec069228257976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.074378 1794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:52:07.075415 1794 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:52:07.075562 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 262464ac31cf4df3b0bb5690a24e53ab (DEFAULT_TABLE table=default.Table_6d7be80f24cc434c949137a402b4b7c7 [id=0c3cfe5b4048494f8cb814f29d799cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.075732 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:52:07.075881 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262464ac31cf4df3b0bb5690a24e53ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.076678 743 tablet_service.cc:1468] Processing CreateTablet for tablet 81f45ab55e414eb39dec069228257976 (DEFAULT_TABLE table=default.Table_f471ebec4d67405899f155aad9d3ec83 [id=35ca80beadf742df969e352ff737586d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.077068 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f45ab55e414eb39dec069228257976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.079574 746 tablet_service.cc:1468] Processing CreateTablet for tablet 35c7bf4272994e9b8a59062b1757f6f5 (DEFAULT_TABLE table=default.Table_e5270cb847db4a57979fd35c13f6fe0b [id=607b89f622c240f0917158cf08a94f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.079653 757 tablet_service.cc:1468] Processing CreateTablet for tablet e1549b9e797a44f89b316a44c5e7ee2e (DEFAULT_TABLE table=default.Table_64d842d3737b421f9832c07a67be66e7 [id=39762a8b3fbd4e489f3da9b819228b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.079888 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c7bf4272994e9b8a59062b1757f6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.081334 893 tablet_service.cc:1468] Processing CreateTablet for tablet e1549b9e797a44f89b316a44c5e7ee2e (DEFAULT_TABLE table=default.Table_64d842d3737b421f9832c07a67be66e7 [id=39762a8b3fbd4e489f3da9b819228b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.081960 884 tablet_service.cc:1468] Processing CreateTablet for tablet 81f45ab55e414eb39dec069228257976 (DEFAULT_TABLE table=default.Table_f471ebec4d67405899f155aad9d3ec83 [id=35ca80beadf742df969e352ff737586d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.082479 892 tablet_service.cc:1468] Processing CreateTablet for tablet 35c7bf4272994e9b8a59062b1757f6f5 (DEFAULT_TABLE table=default.Table_e5270cb847db4a57979fd35c13f6fe0b [id=607b89f622c240f0917158cf08a94f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.081350 888 tablet_service.cc:1468] Processing CreateTablet for tablet 262464ac31cf4df3b0bb5690a24e53ab (DEFAULT_TABLE table=default.Table_6d7be80f24cc434c949137a402b4b7c7 [id=0c3cfe5b4048494f8cb814f29d799cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.083416 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1549b9e797a44f89b316a44c5e7ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.083590 752 tablet_service.cc:1468] Processing CreateTablet for tablet 262464ac31cf4df3b0bb5690a24e53ab (DEFAULT_TABLE table=default.Table_6d7be80f24cc434c949137a402b4b7c7 [id=0c3cfe5b4048494f8cb814f29d799cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.083962 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262464ac31cf4df3b0bb5690a24e53ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.098874 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 664a83b20b344584b5c85db53bfa9021 (DEFAULT_TABLE table=default.Table_42a2e34e5d204e7188555f5e5616cc4c [id=441cf0521bab4fe08b67b987334f492f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.099517 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 664a83b20b344584b5c85db53bfa9021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.101603 741 tablet_service.cc:1468] Processing CreateTablet for tablet 664a83b20b344584b5c85db53bfa9021 (DEFAULT_TABLE table=default.Table_42a2e34e5d204e7188555f5e5616cc4c [id=441cf0521bab4fe08b67b987334f492f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.102025 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 664a83b20b344584b5c85db53bfa9021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.114817 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5788c3143f8c4368a0a765a1827134a9
I20250625 13:52:07.116030 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_419b7a5baa6e4aed809e690190c9f73a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.116636 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_419b7a5baa6e4aed809e690190c9f73a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:07.118096 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 474356bf56b94681a8dc9071c0d8e79a because the tablet was not found. No further retry: Not found: Tablet not found: 474356bf56b94681a8dc9071c0d8e79a
I20250625 13:52:07.118505 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5788c3143f8c4368a0a765a1827134a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.118755 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 5788c3143f8c4368a0a765a1827134a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:07.119098 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5788c3143f8c4368a0a765a1827134a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5788c3143f8c4368a0a765a1827134a9 already in progress: creating tablet
I20250625 13:52:07.119016 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5bc830734d3046dcafc7d1ccc0c5c561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.119688 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bc830734d3046dcafc7d1ccc0c5c561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.122036 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 5788c3143f8c4368a0a765a1827134a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.124859 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2317b24fd74909b133dc5a49898cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.125138 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1549b9e797a44f89b316a44c5e7ee2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.125366 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 664a83b20b344584b5c85db53bfa9021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.125798 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f45ab55e414eb39dec069228257976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.126204 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c7bf4272994e9b8a59062b1757f6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.126646 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262464ac31cf4df3b0bb5690a24e53ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:07.128296 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5788c3143f8c4368a0a765a1827134a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5788c3143f8c4368a0a765a1827134a9 already in progress: creating tablet
I20250625 13:52:07.144712 1168 ts_tablet_manager.cc:1918] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.144855 1168 log.cc:1199] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ae4a105e80c1460cbb21e1bdb9b356b1
I20250625 13:52:07.145148 1168 ts_tablet_manager.cc:1939] T ae4a105e80c1460cbb21e1bdb9b356b1 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:07.145885 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 492) took 5535 ms. Trace:
I20250625 13:52:07.145979 1168 rpcz_store.cc:276] 0625 13:52:01.610435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.610492 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:07.003400 (+5392908us) ts_tablet_manager.cc:1145] Deleting tablet ae4a105e80c1460cbb21e1bdb9b356b1
0625 13:52:07.003408 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.141171 (+137763us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.144592 (+ 3421us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.145867 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5392700}
I20250625 13:52:07.146457 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 493) took 5533 ms. Trace:
I20250625 13:52:07.146566 1168 rpcz_store.cc:276] 0625 13:52:01.613383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.613436 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:07.146302 (+5532866us) ts_tablet_manager.cc:1145] Deleting tablet d85cdadfd694486590092a7b6f61f44c
0625 13:52:07.146314 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.146438 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5532696}
I20250625 13:52:07.146858 1168 tablet_replica.cc:331] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:07.147055 1168 raft_consensus.cc:2241] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
W20250625 13:52:07.147024 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d85cdadfd694486590092a7b6f61f44c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d85cdadfd694486590092a7b6f61f44c already in progress: creating tablet
I20250625 13:52:07.148572 1171 tablet_bootstrap.cc:492] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.148720 1171 ts_tablet_manager.cc:1397] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.179s user 0.001s sys 0.002s
I20250625 13:52:07.149710 1171 raft_consensus.cc:357] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.149866 1171 raft_consensus.cc:383] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.149914 1171 raft_consensus.cc:738] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.150432 1171 consensus_queue.cc:260] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.151006 1171 ts_tablet_manager.cc:1428] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:52:07.151284 1171 tablet_bootstrap.cc:492] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:07.152920 1171 tablet_bootstrap.cc:654] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.155398 1023 tablet_service.cc:1468] Processing CreateTablet for tablet d3e2076007ad48acaef3aa5357299b2e (DEFAULT_TABLE table=default.Table_ff091cba95574357ba76fc2e306f3666 [id=52235c623d184d4ebc54089f44d4a022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.155879 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e2076007ad48acaef3aa5357299b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.160043 1170 tablet_bootstrap.cc:492] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.160215 1170 ts_tablet_manager.cc:1397] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.009s
I20250625 13:52:07.161026 1170 raft_consensus.cc:357] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.161257 1170 raft_consensus.cc:383] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.161351 1170 raft_consensus.cc:738] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.161659 1170 consensus_queue.cc:260] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.162914 1166 tablet_bootstrap.cc:492] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.163446 1166 ts_tablet_manager.cc:1397] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.287s user 0.074s sys 0.040s
I20250625 13:52:07.164129 1170 ts_tablet_manager.cc:1428] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:07.164755 1170 tablet_bootstrap.cc:492] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.165656 1166 raft_consensus.cc:357] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.165999 1166 raft_consensus.cc:383] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.166126 1166 raft_consensus.cc:738] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.166415 1166 consensus_queue.cc:260] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.167153 1170 tablet_bootstrap.cc:654] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.168577 1166 ts_tablet_manager.cc:1428] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:07.169320 892 tablet_service.cc:1468] Processing CreateTablet for tablet d3e2076007ad48acaef3aa5357299b2e (DEFAULT_TABLE table=default.Table_ff091cba95574357ba76fc2e306f3666 [id=52235c623d184d4ebc54089f44d4a022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.169761 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e2076007ad48acaef3aa5357299b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.172151 1166 tablet_bootstrap.cc:492] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.176765 1166 tablet_bootstrap.cc:654] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.180261 757 tablet_service.cc:1468] Processing CreateTablet for tablet d3e2076007ad48acaef3aa5357299b2e (DEFAULT_TABLE table=default.Table_ff091cba95574357ba76fc2e306f3666 [id=52235c623d184d4ebc54089f44d4a022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.180754 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e2076007ad48acaef3aa5357299b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.185154 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bdbf586b82194a24a2259cbeb62d615c
I20250625 13:52:07.185698 591 catalog_manager.cc:6652] ReplaceTablet: tablet bdbf586b82194a24a2259cbeb62d615c deleted and replaced by tablet f5203c047f514dbcaf57087dd8c3e2b6
I20250625 13:52:07.186254 1166 tablet_bootstrap.cc:492] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.186373 1166 ts_tablet_manager.cc:1397] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20250625 13:52:07.187440 889 tablet_service.cc:1515] Processing DeleteTablet for tablet bdbf586b82194a24a2259cbeb62d615c with delete_type TABLET_DATA_DELETED (replaced by tablet f5203c047f514dbcaf57087dd8c3e2b6) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.187057 1166 raft_consensus.cc:357] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.187798 1166 raft_consensus.cc:383] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.187911 1166 raft_consensus.cc:738] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.188181 1166 consensus_queue.cc:260] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.192516 1166 ts_tablet_manager.cc:1428] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:07.192904 752 tablet_service.cc:1468] Processing CreateTablet for tablet e9a5bcf4a7024ee39349358b15c7379b (DEFAULT_TABLE table=default.Table_d373cb07af7a453dad8f774642a336dd [id=fe455b3fa15e4c7887e4d0a4ecbefd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.193409 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a5bcf4a7024ee39349358b15c7379b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.194054 892 tablet_service.cc:1468] Processing CreateTablet for tablet e9a5bcf4a7024ee39349358b15c7379b (DEFAULT_TABLE table=default.Table_d373cb07af7a453dad8f774642a336dd [id=fe455b3fa15e4c7887e4d0a4ecbefd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.194445 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a5bcf4a7024ee39349358b15c7379b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.196887 1837 raft_consensus.cc:491] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.197068 1837 raft_consensus.cc:513] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.197075 1014 tablet_service.cc:1468] Processing CreateTablet for tablet e9a5bcf4a7024ee39349358b15c7379b (DEFAULT_TABLE table=default.Table_d373cb07af7a453dad8f774642a336dd [id=fe455b3fa15e4c7887e4d0a4ecbefd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.197702 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a5bcf4a7024ee39349358b15c7379b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.197795 1837 leader_election.cc:290] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:07.198724 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abd6f9121a9d4b39adde1b880a875e23" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:07.199646 849 leader_election.cc:343] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:07.201414 851 leader_election.cc:343] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: abd6f9121a9d4b39adde1b880a875e23
I20250625 13:52:07.201545 851 leader_election.cc:304] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:07.202155 1837 raft_consensus.cc:2747] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:07.203874 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 5565f43d293e4e809d6aff72a201db3e (DEFAULT_TABLE table=default.Table_50bfa9943b744384b9b3911caaf604b3 [id=b85c65b44b9d4c258ca6d788885202b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.204344 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5565f43d293e4e809d6aff72a201db3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.205056 889 tablet_service.cc:1468] Processing CreateTablet for tablet 5565f43d293e4e809d6aff72a201db3e (DEFAULT_TABLE table=default.Table_50bfa9943b744384b9b3911caaf604b3 [id=b85c65b44b9d4c258ca6d788885202b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.205441 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5565f43d293e4e809d6aff72a201db3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.206802 1166 tablet_bootstrap.cc:492] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.207350 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 8a4adf5c25a04a41895e9c35f57e94b2 (DEFAULT_TABLE table=default.Table_4fcd474fe7fd48c0b7a71988682079c4 [id=80fc1c40f8e045c990ad24d6d10b3c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.207522 1015 tablet_service.cc:1468] Processing CreateTablet for tablet abdd0ef05eb04dc3b9925076bab0b5de (DEFAULT_TABLE table=default.Table_6b02faeef1a54d03b3fedf55f1bbbc8a [id=277abdfbb8fe4ab4a8cea23b2f0849f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.207749 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4adf5c25a04a41895e9c35f57e94b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.208343 1166 tablet_bootstrap.cc:654] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.209069 741 tablet_service.cc:1515] Processing DeleteTablet for tablet bdbf586b82194a24a2259cbeb62d615c with delete_type TABLET_DATA_DELETED (replaced by tablet f5203c047f514dbcaf57087dd8c3e2b6) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.211669 893 tablet_service.cc:1468] Processing CreateTablet for tablet abdd0ef05eb04dc3b9925076bab0b5de (DEFAULT_TABLE table=default.Table_6b02faeef1a54d03b3fedf55f1bbbc8a [id=277abdfbb8fe4ab4a8cea23b2f0849f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.212123 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdd0ef05eb04dc3b9925076bab0b5de. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:07.212975 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bdbf586b82194a24a2259cbeb62d615c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdbf586b82194a24a2259cbeb62d615c already in progress: creating tablet
I20250625 13:52:07.213420 752 tablet_service.cc:1468] Processing CreateTablet for tablet abdd0ef05eb04dc3b9925076bab0b5de (DEFAULT_TABLE table=default.Table_6b02faeef1a54d03b3fedf55f1bbbc8a [id=277abdfbb8fe4ab4a8cea23b2f0849f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.213776 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdd0ef05eb04dc3b9925076bab0b5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.214258 1171 tablet_bootstrap.cc:492] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.214375 1171 ts_tablet_manager.cc:1397] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.063s user 0.000s sys 0.003s
I20250625 13:52:07.215826 1171 raft_consensus.cc:357] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.216073 1171 raft_consensus.cc:383] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.216161 1171 raft_consensus.cc:738] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.216424 1171 consensus_queue.cc:260] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.216899 1171 ts_tablet_manager.cc:1428] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250625 13:52:07.217171 1171 tablet_bootstrap.cc:492] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:07.231355 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8a4adf5c25a04a41895e9c35f57e94b2 (DEFAULT_TABLE table=default.Table_4fcd474fe7fd48c0b7a71988682079c4 [id=80fc1c40f8e045c990ad24d6d10b3c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.231904 746 tablet_service.cc:1468] Processing CreateTablet for tablet c0417fae218a4ec8bfd18f4834a0789f (DEFAULT_TABLE table=default.Table_bd76ad95ad844003ad3afa39db18c7f8 [id=edd692bcb0914ed89d5daf5f5d73d285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.232348 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4adf5c25a04a41895e9c35f57e94b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.232434 743 tablet_service.cc:1468] Processing CreateTablet for tablet 7221e009bda64b769846af1eb861d4fc (DEFAULT_TABLE table=default.Table_5c1dbe2671ff41d9b8d115b4b7ea58d6 [id=e2779d18910847aa878f92c43845d557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.232728 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7221e009bda64b769846af1eb861d4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.233050 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0417fae218a4ec8bfd18f4834a0789f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.238967 884 tablet_service.cc:1468] Processing CreateTablet for tablet c0417fae218a4ec8bfd18f4834a0789f (DEFAULT_TABLE table=default.Table_bd76ad95ad844003ad3afa39db18c7f8 [id=edd692bcb0914ed89d5daf5f5d73d285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.239369 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0417fae218a4ec8bfd18f4834a0789f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.244467 888 tablet_service.cc:1468] Processing CreateTablet for tablet 7221e009bda64b769846af1eb861d4fc (DEFAULT_TABLE table=default.Table_5c1dbe2671ff41d9b8d115b4b7ea58d6 [id=e2779d18910847aa878f92c43845d557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.244926 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7221e009bda64b769846af1eb861d4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.249912 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 7221e009bda64b769846af1eb861d4fc (DEFAULT_TABLE table=default.Table_5c1dbe2671ff41d9b8d115b4b7ea58d6 [id=e2779d18910847aa878f92c43845d557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.250658 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7221e009bda64b769846af1eb861d4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.253758 741 tablet_service.cc:1468] Processing CreateTablet for tablet 5565f43d293e4e809d6aff72a201db3e (DEFAULT_TABLE table=default.Table_50bfa9943b744384b9b3911caaf604b3 [id=b85c65b44b9d4c258ca6d788885202b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.254175 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5565f43d293e4e809d6aff72a201db3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.255738 1014 tablet_service.cc:1468] Processing CreateTablet for tablet c0417fae218a4ec8bfd18f4834a0789f (DEFAULT_TABLE table=default.Table_bd76ad95ad844003ad3afa39db18c7f8 [id=edd692bcb0914ed89d5daf5f5d73d285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.256103 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0417fae218a4ec8bfd18f4834a0789f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.259529 1168 raft_consensus.cc:2270] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.260149 1168 ts_tablet_manager.cc:1905] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.260296 1170 tablet_bootstrap.cc:492] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.260462 1170 ts_tablet_manager.cc:1397] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.096s user 0.000s sys 0.003s
I20250625 13:52:07.261469 1170 raft_consensus.cc:357] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.261696 1170 raft_consensus.cc:383] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.261783 1170 raft_consensus.cc:738] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.262099 1170 consensus_queue.cc:260] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.262293 892 tablet_service.cc:1468] Processing CreateTablet for tablet 8a4adf5c25a04a41895e9c35f57e94b2 (DEFAULT_TABLE table=default.Table_4fcd474fe7fd48c0b7a71988682079c4 [id=80fc1c40f8e045c990ad24d6d10b3c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.263556 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4adf5c25a04a41895e9c35f57e94b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.264226 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet bdbf586b82194a24a2259cbeb62d615c with delete_type TABLET_DATA_DELETED (replaced by tablet f5203c047f514dbcaf57087dd8c3e2b6) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.264287 1171 tablet_bootstrap.cc:654] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.265635 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3cf16642852d4afa96c4425698093888"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.265941 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bdbf586b82194a24a2259cbeb62d615c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdbf586b82194a24a2259cbeb62d615c already in progress: creating tablet
W20250625 13:52:07.266212 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cf16642852d4afa96c4425698093888 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.271191 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abdd0ef05eb04dc3b9925076bab0b5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.272285 881 tablet_service.cc:1468] Processing CreateTablet for tablet 6110bcce59ea452cbc6e9f464483bde9 (DEFAULT_TABLE table=default.Table_f4fee43fefb74cd5bd6e7cc25308e969 [id=f8a558fc0cff454ea7f53c17be4533cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.272292 1023 tablet_service.cc:1468] Processing CreateTablet for tablet c7ad5e90188847f38894fc3c1f8f2693 (DEFAULT_TABLE table=default.Table_0b1631a468c34b49a62449290f63ea32 [id=116969aa3f7a4ef29b7007b14e1deddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.272513 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 474356bf56b94681a8dc9071c0d8e79a (DEFAULT_TABLE table=default.Table_9d8551fdf4e043cd90e2ce8228c3db7c [id=4e7af00c727c43debb87a97e832325e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.272644 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6110bcce59ea452cbc6e9f464483bde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.272704 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ad5e90188847f38894fc3c1f8f2693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.272984 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 62f748645e894531b3b1eb694d45e6b0 (DEFAULT_TABLE table=default.Table_945803b681074611b2bfcd213a5e072d [id=19abba73a8f34966afef3923120eda5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.273283 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f748645e894531b3b1eb694d45e6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.273537 884 tablet_service.cc:1468] Processing CreateTablet for tablet 62f748645e894531b3b1eb694d45e6b0 (DEFAULT_TABLE table=default.Table_945803b681074611b2bfcd213a5e072d [id=19abba73a8f34966afef3923120eda5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.273855 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f748645e894531b3b1eb694d45e6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.275555 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6110bcce59ea452cbc6e9f464483bde9 (DEFAULT_TABLE table=default.Table_f4fee43fefb74cd5bd6e7cc25308e969 [id=f8a558fc0cff454ea7f53c17be4533cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.275875 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6110bcce59ea452cbc6e9f464483bde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.278861 1166 tablet_bootstrap.cc:492] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.279004 1166 ts_tablet_manager.cc:1397] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.072s user 0.033s sys 0.020s
I20250625 13:52:07.279492 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 474356bf56b94681a8dc9071c0d8e79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.279740 1166 raft_consensus.cc:357] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.279922 1166 raft_consensus.cc:383] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.279970 1166 raft_consensus.cc:738] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.280303 1166 consensus_queue.cc:260] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.282266 1166 ts_tablet_manager.cc:1428] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:07.282560 1166 tablet_bootstrap.cc:492] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.284264 1166 tablet_bootstrap.cc:654] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.287595 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f1d647c2504c494d8b60c751126ee825
I20250625 13:52:07.288341 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_21e1c6c522e04d72bb8739a9b7ef211e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.288926 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21e1c6c522e04d72bb8739a9b7ef211e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.288910 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet f1d647c2504c494d8b60c751126ee825 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:07.289580 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f1d647c2504c494d8b60c751126ee825 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1d647c2504c494d8b60c751126ee825 already in progress: creating tablet
I20250625 13:52:07.289716 881 tablet_service.cc:1515] Processing DeleteTablet for tablet f1d647c2504c494d8b60c751126ee825 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.290095 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76
I20250625 13:52:07.290552 594 catalog_manager.cc:6652] ReplaceTablet: tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 deleted and replaced by tablet 2b29392cfeb14f46a60fd0731f220127
I20250625 13:52:07.290938 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f49bf03b78c54f3eabcb74b4211ae036" } new_table_name: "default.Table_6d9870087cdc4dfca8fa438020887d80" modify_external_catalogs: true
I20250625 13:52:07.291194 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b29392cfeb14f46a60fd0731f220127) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.291625 881 tablet_service.cc:1468] Processing CreateTablet for tablet 879d91dcfb0b4868a7ba6833c211e5eb (DEFAULT_TABLE table=default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 [id=5b767eccaaaa46e0a054dd083e181cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.292663 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 879d91dcfb0b4868a7ba6833c211e5eb (DEFAULT_TABLE table=default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 [id=5b767eccaaaa46e0a054dd083e181cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.292690 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_87f21414bdc44bddaaa231cd2dfc7ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:07.293025 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879d91dcfb0b4868a7ba6833c211e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:07.293138 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87f21414bdc44bddaaa231cd2dfc7ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.295814 593 master_service.cc:948] ReplaceTablet: received request to replace tablet d43ba17ebfec4244894d1b6e23200c33 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:07.296326 593 master_service.cc:948] ReplaceTablet: received request to replace tablet e99fe5ad690049029b8325dd38a11580 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:07.297473 884 tablet_service.cc:1515] Processing DeleteTablet for tablet abd6f9121a9d4b39adde1b880a875e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.297636 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd51cad7d6342d7a1d45f59653fa32f) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.297654 892 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:07.297755 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 529c3bf8e72f46948cfb864445ff51f3 consensus state has no leader
I20250625 13:52:07.298195 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a443a849fceb423c9b889f5c3c69b2b6" } modify_external_catalogs: true
I20250625 13:52:07.298388 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a443a849fceb423c9b889f5c3c69b2b6" } modify_external_catalogs: true
I20250625 13:52:07.300382 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:07.301311 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_44a75b53cab0470da32c4aedecdc2434" } modify_external_catalogs: true
I20250625 13:52:07.301822 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:07.302696 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b1982b3d458a4586a464994f13dd1481" } new_table_name: "default.Table_6cc68a263f834052932d28d12db66d33" modify_external_catalogs: true
I20250625 13:52:07.304795 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_89f74db6137d499bbb022bd99ebc4243" } modify_external_catalogs: true
I20250625 13:52:07.304989 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_89f74db6137d499bbb022bd99ebc4243" } modify_external_catalogs: true
I20250625 13:52:07.307158 1776 raft_consensus.cc:491] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.307343 1776 raft_consensus.cc:513] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.308243 1776 leader_election.cc:290] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.308241 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b29392cfeb14f46a60fd0731f220127) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:07.308861 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 already in progress: creating tablet
I20250625 13:52:07.308874 752 tablet_service.cc:1468] Processing CreateTablet for tablet 474356bf56b94681a8dc9071c0d8e79a (DEFAULT_TABLE table=default.Table_9d8551fdf4e043cd90e2ce8228c3db7c [id=4e7af00c727c43debb87a97e832325e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.309139 746 tablet_service.cc:1468] Processing CreateTablet for tablet abc44a41b30e4a53977ec81d6905612f (DEFAULT_TABLE table=default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.309211 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_44a75b53cab0470da32c4aedecdc2434" } modify_external_catalogs: true
I20250625 13:52:07.309455 757 tablet_service.cc:1468] Processing CreateTablet for tablet 62f748645e894531b3b1eb694d45e6b0 (DEFAULT_TABLE table=default.Table_945803b681074611b2bfcd213a5e072d [id=19abba73a8f34966afef3923120eda5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.310086 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c50e1ebb9042ef8f0a651ba7186b74" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:07.310560 741 tablet_service.cc:1468] Processing CreateTablet for tablet c7ad5e90188847f38894fc3c1f8f2693 (DEFAULT_TABLE table=default.Table_0b1631a468c34b49a62449290f63ea32 [id=116969aa3f7a4ef29b7007b14e1deddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.309229 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 474356bf56b94681a8dc9071c0d8e79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.310876 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ad5e90188847f38894fc3c1f8f2693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.313906 744 tablet_service.cc:1515] Processing DeleteTablet for tablet f1d647c2504c494d8b60c751126ee825 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.314355 751 tablet_service.cc:1468] Processing CreateTablet for tablet 879d91dcfb0b4868a7ba6833c211e5eb (DEFAULT_TABLE table=default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 [id=5b767eccaaaa46e0a054dd083e181cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.314679 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879d91dcfb0b4868a7ba6833c211e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.315433 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b29392cfeb14f46a60fd0731f220127) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.315439 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.315443 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f748645e894531b3b1eb694d45e6b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:07.315832 711 leader_election.cc:343] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.316031 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc44a41b30e4a53977ec81d6905612f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.316740 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879d91dcfb0b4868a7ba6833c211e5eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.323730 1776 raft_consensus.cc:491] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.323915 1776 raft_consensus.cc:513] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.324628 1776 leader_election.cc:290] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.324954 743 tablet_service.cc:1468] Processing CreateTablet for tablet 6110bcce59ea452cbc6e9f464483bde9 (DEFAULT_TABLE table=default.Table_f4fee43fefb74cd5bd6e7cc25308e969 [id=f8a558fc0cff454ea7f53c17be4533cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.325358 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6110bcce59ea452cbc6e9f464483bde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.325471 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04dd85668a47461e9e82f8b251468ad3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:07.326037 711 leader_election.cc:343] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.326574 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c50e1ebb9042ef8f0a651ba7186b74" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:07.326668 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04dd85668a47461e9e82f8b251468ad3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:07.327164 711 leader_election.cc:343] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.327275 711 leader_election.cc:304] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:07.327387 1171 tablet_bootstrap.cc:492] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.327525 1171 ts_tablet_manager.cc:1397] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.000s
W20250625 13:52:07.327674 711 leader_election.cc:343] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.327805 711 leader_election.cc:304] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:07.328189 1776 raft_consensus.cc:2747] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:07.328338 1776 raft_consensus.cc:2747] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:07.328703 1171 raft_consensus.cc:357] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.328918 1171 raft_consensus.cc:383] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.329021 1171 raft_consensus.cc:738] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.329286 1171 consensus_queue.cc:260] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.332214 1171 ts_tablet_manager.cc:1428] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:52:07.332548 1171 tablet_bootstrap.cc:492] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:07.341127 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f1d647c2504c494d8b60c751126ee825 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1d647c2504c494d8b60c751126ee825 already in progress: creating tablet
W20250625 13:52:07.341629 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 already in progress: creating tablet
W20250625 13:52:07.342489 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4fa801f7d98e4794b6ca439638f29e69 because the tablet was not found. No further retry: Not found: Tablet not found: 4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:07.353350 1171 tablet_bootstrap.cc:654] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:07.357851 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38770320fc6541c39e10448978f8a3b0 consensus state has no leader
I20250625 13:52:07.358330 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_103d81d6145e465b96d0044d4fd3d4f5" } new_table_name: "default.Table_7f21a5a5b40740a28dd60ad19524d6e4" modify_external_catalogs: true
I20250625 13:52:07.358842 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 29be8c0350584de382985e66556bdcca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.359089 1710 tablet_replica.cc:331] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:07.359246 1710 raft_consensus.cc:2241] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.359477 1710 raft_consensus.cc:2270] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.360107 1710 ts_tablet_manager.cc:1905] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.361975 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e99fe5ad690049029b8325dd38a11580
I20250625 13:52:07.363431 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet e99fe5ad690049029b8325dd38a11580 with delete_type TABLET_DATA_DELETED (replaced by tablet d2e8f4e3b45a4065b74485157f31247d) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.364185 881 tablet_service.cc:1515] Processing DeleteTablet for tablet e99fe5ad690049029b8325dd38a11580 with delete_type TABLET_DATA_DELETED (replaced by tablet d2e8f4e3b45a4065b74485157f31247d) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.364570 593 catalog_manager.cc:6652] ReplaceTablet: tablet e99fe5ad690049029b8325dd38a11580 deleted and replaced by tablet d2e8f4e3b45a4065b74485157f31247d
I20250625 13:52:07.351637 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5821b12a3aff40dcb22e592583f8c1f5" } new_table_name: "default.Table_1b6dba3c0b1f4e9da6c17ca8017e4b93" modify_external_catalogs: true
I20250625 13:52:07.365115 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a0aeb065271449cb97100a02d36b89db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:07.366458 888 tablet_service.cc:1468] Processing CreateTablet for tablet abc44a41b30e4a53977ec81d6905612f (DEFAULT_TABLE table=default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.368093 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 11711546f0e14f9a9eabf003f49a69b7 with delete_type TABLET_DATA_DELETED (replaced by tablet c605d5b64c164c1594b0a40cfa2c4cf6) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.368297 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 2cd5303e674246ada7d36574b535b333 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.369210 596 master_service.cc:948] ReplaceTablet: received request to replace tablet d2972d4366c143498f746fd62a039acd from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:07.370157 751 tablet_service.cc:1515] Processing DeleteTablet for tablet e99fe5ad690049029b8325dd38a11580 with delete_type TABLET_DATA_DELETED (replaced by tablet d2e8f4e3b45a4065b74485157f31247d) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.370966 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_73b9f78abc404d3c98a2a57d96a30f84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:07.375090 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d2972d4366c143498f746fd62a039acd
I20250625 13:52:07.377007 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ef2c3757c0284aaca70ec317c05195d6
I20250625 13:52:07.378566 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet d2972d4366c143498f746fd62a039acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.378890 595 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet bf009e228f7a4cffa8707a31e240c42c
I20250625 13:52:07.379261 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ef2c3757c0284aaca70ec317c05195d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:07.379796 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e99fe5ad690049029b8325dd38a11580 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e99fe5ad690049029b8325dd38a11580 already in progress: creating tablet
I20250625 13:52:07.380457 893 tablet_service.cc:1468] Processing CreateTablet for tablet c7ad5e90188847f38894fc3c1f8f2693 (DEFAULT_TABLE table=default.Table_0b1631a468c34b49a62449290f63ea32 [id=116969aa3f7a4ef29b7007b14e1deddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.380888 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ef2c3757c0284aaca70ec317c05195d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:07.381476 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0aeb065271449cb97100a02d36b89db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.383378 889 tablet_service.cc:1468] Processing CreateTablet for tablet 474356bf56b94681a8dc9071c0d8e79a (DEFAULT_TABLE table=default.Table_9d8551fdf4e043cd90e2ce8228c3db7c [id=4e7af00c727c43debb87a97e832325e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.386313 1019 tablet_service.cc:1468] Processing CreateTablet for tablet abc44a41b30e4a53977ec81d6905612f (DEFAULT_TABLE table=default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:07.392005 751 tablet_service.cc:1515] Processing DeleteTablet for tablet d2972d4366c143498f746fd62a039acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.392743 1821 raft_consensus.cc:491] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.392880 1821 raft_consensus.cc:513] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.393498 1821 leader_election.cc:290] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.394153 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:07.395087 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f72c555d9842f28ca4bf310da9c53c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:07.395597 983 leader_election.cc:304] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:07.396224 1821 raft_consensus.cc:2747] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:07.396622 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ef2c3757c0284aaca70ec317c05195d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef2c3757c0284aaca70ec317c05195d6 already in progress: creating tablet
I20250625 13:52:07.399994 881 tablet_service.cc:1515] Processing DeleteTablet for tablet d2972d4366c143498f746fd62a039acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.399967 892 tablet_service.cc:1515] Processing DeleteTablet for tablet ef2c3757c0284aaca70ec317c05195d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:07.400542 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d2972d4366c143498f746fd62a039acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2972d4366c143498f746fd62a039acd already in progress: creating tablet
I20250625 13:52:07.402504 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9bfb4aea9df24975a81b278ceadf7542"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.403343 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bfb4aea9df24975a81b278ceadf7542 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.400557 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_791629af534b4515bda726fca5095041"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.406687 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9ae350a9dbf476081dc56a709de42c8 has no consensus state
W20250625 13:52:07.410554 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bee9cd33bff04a69ac66ba23d9577603 consensus state has no leader
W20250625 13:52:07.411607 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_791629af534b4515bda726fca5095041 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.412096 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3c5334740bcc46cdb576090f3911074a" } modify_external_catalogs: true
I20250625 13:52:07.412452 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:07.414883 777 raft_consensus.cc:1273] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer affa1c3035b940e88ca143be7d35840d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:07.414960 1170 ts_tablet_manager.cc:1428] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.154s user 0.001s sys 0.002s
I20250625 13:52:07.415495 1510 consensus_queue.cc:1035] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:52:07.415724 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73b9f78abc404d3c98a2a57d96a30f84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:07.416379 849 consensus_peers.cc:487] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:07.416462 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3c5334740bcc46cdb576090f3911074a" } modify_external_catalogs: true
W20250625 13:52:07.417441 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d9bd351536c4558b189cebf8b793343 consensus state has no leader
I20250625 13:52:07.418614 1821 raft_consensus.cc:491] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.418781 1821 raft_consensus.cc:513] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.419415 1821 leader_election.cc:290] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.419942 1170 tablet_bootstrap.cc:492] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:07.420208 985 leader_election.cc:343] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 11711546f0e14f9a9eabf003f49a69b7
I20250625 13:52:07.421619 1170 tablet_bootstrap.cc:654] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.423557 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc44a41b30e4a53977ec81d6905612f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.424288 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 474356bf56b94681a8dc9071c0d8e79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.424862 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ad5e90188847f38894fc3c1f8f2693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.427260 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 1effed0fa10d409abcb31dbba54861d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.428122 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11711546f0e14f9a9eabf003f49a69b7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:07.428318 915 raft_consensus.cc:2466] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:07.428970 983 leader_election.cc:304] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:07.429421 1821 raft_consensus.cc:2802] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:07.429513 1821 raft_consensus.cc:491] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:07.429579 1821 raft_consensus.cc:3058] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:07.433811 2023 tablet.cc:1722] T 88f72c555d9842f28ca4bf310da9c53c P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:07.423584 1168 ts_tablet_manager.cc:1918] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:07.436411 1168 log.cc:1199] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/15e58fbec94e4ff9b7ac9830b576fe7f
I20250625 13:52:07.436944 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e7bd4939714a4359846345ed0521701c
I20250625 13:52:07.437034 1168 ts_tablet_manager.cc:1939] T 15e58fbec94e4ff9b7ac9830b576fe7f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:07.438421 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet e7bd4939714a4359846345ed0521701c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.438808 892 tablet_service.cc:1515] Processing DeleteTablet for tablet e7bd4939714a4359846345ed0521701c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.440019 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abc44a41b30e4a53977ec81d6905612f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:07.440989 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e7bd4939714a4359846345ed0521701c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.441378 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 653267b468ef4958b98af0ed6a7bf4d3 (table default.Table_107a64d05b814c5a9cc7b956f38af315 [id=6bfad46432b7473f88d77bfcfa2a4e9a]). Expected version 2 got 0
W20250625 13:52:07.442015 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e7bd4939714a4359846345ed0521701c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7bd4939714a4359846345ed0521701c already in progress: creating tablet
I20250625 13:52:07.453094 2013 tablet.cc:1722] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:07.454234 2014 tablet.cc:1722] T 88f72c555d9842f28ca4bf310da9c53c P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:07.459672 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:07.460094 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:07.461283 1171 tablet_bootstrap.cc:492] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.461427 1171 ts_tablet_manager.cc:1397] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.129s user 0.005s sys 0.002s
I20250625 13:52:07.462798 743 tablet_service.cc:1515] Processing DeleteTablet for tablet a0e81d4c904248eba858462bcf31c5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.462800 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.463112 1165 tablet_replica.cc:331] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:07.463286 1165 raft_consensus.cc:2241] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.463464 1165 raft_consensus.cc:2270] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.463371 1171 raft_consensus.cc:357] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.463532 1171 raft_consensus.cc:383] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.464210 1165 ts_tablet_manager.cc:1905] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.463588 1171 raft_consensus.cc:738] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.467417 1171 consensus_queue.cc:260] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.467417 2024 tablet.cc:1722] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:07.470180 1171 ts_tablet_manager.cc:1428] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250625 13:52:07.472152 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:07.473335 1171 tablet_bootstrap.cc:492] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:07.475605 1171 tablet_bootstrap.cc:654] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.477560 1170 tablet_bootstrap.cc:492] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.477696 1170 ts_tablet_manager.cc:1397] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.002s
I20250625 13:52:07.478302 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:07.478438 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 653267b468ef4958b98af0ed6a7bf4d3 (table default.Table_107a64d05b814c5a9cc7b956f38af315 [id=6bfad46432b7473f88d77bfcfa2a4e9a]). Expected version 2 got 0
I20250625 13:52:07.483776 1166 tablet_bootstrap.cc:492] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.483984 1166 ts_tablet_manager.cc:1397] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.201s user 0.039s sys 0.037s
I20250625 13:52:07.484712 1166 raft_consensus.cc:357] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.484886 1166 raft_consensus.cc:383] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.484944 1166 raft_consensus.cc:738] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.485181 1166 consensus_queue.cc:260] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.485664 1166 ts_tablet_manager.cc:1428] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:07.482645 1170 raft_consensus.cc:357] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.485950 1170 raft_consensus.cc:383] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.486029 1170 raft_consensus.cc:738] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.486241 1170 consensus_queue.cc:260] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.486809 1166 tablet_bootstrap.cc:492] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.488333 1166 tablet_bootstrap.cc:654] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.491077 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2d095d613a32493bbedabb482c8f7d88" } modify_external_catalogs: true
I20250625 13:52:07.491295 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2d095d613a32493bbedabb482c8f7d88" } modify_external_catalogs: true
I20250625 13:52:07.494319 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 494) took 5862 ms. Trace:
I20250625 13:52:07.494428 1168 rpcz_store.cc:276] 0625 13:52:01.631966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.632021 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:07.146814 (+5514793us) ts_tablet_manager.cc:1145] Deleting tablet 15e58fbec94e4ff9b7ac9830b576fe7f
0625 13:52:07.146825 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.322625 (+175800us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.423464 (+100839us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.494293 (+ 70829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5514576}
I20250625 13:52:07.494921 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 495) took 5852 ms. Trace:
I20250625 13:52:07.495002 1168 rpcz_store.cc:276] 0625 13:52:01.641980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.642041 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:07.494784 (+5852743us) ts_tablet_manager.cc:1145] Deleting tablet 4ae5f1d414794474b2178b431cbc1d44
0625 13:52:07.494797 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.494902 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5852542}
I20250625 13:52:07.495129 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 15e58fbec94e4ff9b7ac9830b576fe7f (table default.Table_72b53bc133ea42568572d075e5493c2e [id=df6a0a5985da4e6e82eb4b99149e24ba]) successfully deleted
I20250625 13:52:07.495401 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 496) took 5802 ms. Trace:
I20250625 13:52:07.495478 1168 rpcz_store.cc:276] 0625 13:52:01.693379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.693455 (+ 76us) service_pool.cc:225] Handling call
0625 13:52:07.495257 (+5801802us) ts_tablet_manager.cc:1145] Deleting tablet 17b172c7631048ceb97ccbea896f7f5b
0625 13:52:07.495266 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.495384 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5798081}
I20250625 13:52:07.495795 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 497) took 5757 ms. Trace:
I20250625 13:52:07.495867 1168 rpcz_store.cc:276] 0625 13:52:01.738392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.738454 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:07.495691 (+5757237us) ts_tablet_manager.cc:1145] Deleting tablet a29708fb688a4a65b84ae2564a040b4c
0625 13:52:07.495700 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.495781 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5757008}
I20250625 13:52:07.496111 1168 tablet_replica.cc:331] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:07.496312 1168 raft_consensus.cc:2241] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:07.496663 1168 raft_consensus.cc:2270] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.497284 1168 ts_tablet_manager.cc:1905] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:07.498271 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4ae5f1d414794474b2178b431cbc1d44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ae5f1d414794474b2178b431cbc1d44 already in progress: creating tablet
W20250625 13:52:07.498708 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 17b172c7631048ceb97ccbea896f7f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17b172c7631048ceb97ccbea896f7f5b already in progress: creating tablet
I20250625 13:52:07.505932 743 tablet_service.cc:1515] Processing DeleteTablet for tablet d43ba17ebfec4244894d1b6e23200c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.510370 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6cc68a263f834052932d28d12db66d33" } new_table_name: "default.Table_6a309ccf2fff4b8fae82108cf1b95886" modify_external_catalogs: true
I20250625 13:52:07.519629 1821 raft_consensus.cc:513] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.520294 1821 leader_election.cc:290] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.520813 1710 ts_tablet_manager.cc:1918] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.520906 1710 log.cc:1199] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/29be8c0350584de382985e66556bdcca
I20250625 13:52:07.521170 1710 ts_tablet_manager.cc:1939] T 29be8c0350584de382985e66556bdcca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.525482 1842 raft_consensus.cc:491] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.525660 1842 raft_consensus.cc:513] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.526539 1842 leader_election.cc:290] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:07.527493 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d89b0ddcc14148faa1a6926fcbb027a8
W20250625 13:52:07.528281 849 leader_election.cc:343] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 04479dfb34584f0891c604ef72bf93b5
I20250625 13:52:07.528858 743 tablet_service.cc:1515] Processing DeleteTablet for tablet d89b0ddcc14148faa1a6926fcbb027a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:07.529366 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet bee9cd33bff04a69ac66ba23d9577603 (table default.Table_6cc68a263f834052932d28d12db66d33 [id=4055346986414eec9af02674c2dc1628], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bee9cd33bff04a69ac66ba23d9577603 consensus state has no leader
I20250625 13:52:07.529860 888 tablet_service.cc:1515] Processing DeleteTablet for tablet d89b0ddcc14148faa1a6926fcbb027a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.530169 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d89b0ddcc14148faa1a6926fcbb027a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.531932 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e3977ec02e244d9c8c7b932dbf431c53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.532372 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bee9cd33bff04a69ac66ba23d9577603 consensus state has no leader
W20250625 13:52:07.532618 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3977ec02e244d9c8c7b932dbf431c53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.536048 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11711546f0e14f9a9eabf003f49a69b7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:07.536213 915 raft_consensus.cc:3058] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:07.536453 851 leader_election.cc:343] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 04479dfb34584f0891c604ef72bf93b5
I20250625 13:52:07.536578 851 leader_election.cc:304] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:07.537024 1842 raft_consensus.cc:2747] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 13:52:07.542804 985 leader_election.cc:343] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 11711546f0e14f9a9eabf003f49a69b7
I20250625 13:52:07.546268 2018 raft_consensus.cc:491] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.546437 2018 raft_consensus.cc:513] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.547128 2018 leader_election.cc:290] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.547863 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf73b4ab99c466abcfedf1b200e743b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:07.548009 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf73b4ab99c466abcfedf1b200e743b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:07.548651 711 leader_election.cc:343] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:07.548864 711 leader_election.cc:343] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.548952 711 leader_election.cc:304] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:07.549337 2018 raft_consensus.cc:2747] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:07.553416 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7b7294492237439b955837e068dea004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.554097 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b7294492237439b955837e068dea004 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.556460 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6277ec7df0c84c3b9af326711152e888"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.557148 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6277ec7df0c84c3b9af326711152e888 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.558485 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.559763 1821 raft_consensus.cc:491] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.559919 1821 raft_consensus.cc:513] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.560628 1821 leader_election.cc:290] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:07.561862 985 leader_election.cc:343] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f065b4e3218b4e6e8a2ac3a70eede13b
I20250625 13:52:07.562973 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f065b4e3218b4e6e8a2ac3a70eede13b" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:07.563189 912 raft_consensus.cc:2466] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:07.563660 983 leader_election.cc:304] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:07.564204 1821 raft_consensus.cc:2802] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:07.564291 1821 raft_consensus.cc:491] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:07.564363 1821 raft_consensus.cc:3058] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:07.564488 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e54b0d212c3542cc8cb5fb2e997f6be3" } new_table_name: "default.Table_53926d5babb045a4854877f904b639ec" modify_external_catalogs: true
I20250625 13:52:07.564416 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1b641ce25e2e4126bd10bcd3d7f8386c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.565594 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b641ce25e2e4126bd10bcd3d7f8386c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.559355 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a24126ce121148d29172d84ff249105f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.579440 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a24126ce121148d29172d84ff249105f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.584738 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_af5d40e690b54d16b4689c880fc20261" } modify_external_catalogs: true
I20250625 13:52:07.585021 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_af5d40e690b54d16b4689c880fc20261" } modify_external_catalogs: true
I20250625 13:52:07.586124 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e96b4c55e0434858b439e40c14a99aca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.587402 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e96b4c55e0434858b439e40c14a99aca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:07.598567 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 54997.9us
I20250625 13:52:07.598938 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_58c91d644ba94ed580ae053357892c7a" } modify_external_catalogs: true
I20250625 13:52:07.599170 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_58c91d644ba94ed580ae053357892c7a" } modify_external_catalogs: true
I20250625 13:52:07.602907 1821 raft_consensus.cc:513] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.603804 1821 leader_election.cc:290] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:07.604605 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f065b4e3218b4e6e8a2ac3a70eede13b" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:07.604930 912 raft_consensus.cc:3058] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:07.605404 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0f82cde67a844d2dadb19c95b952db96" } modify_external_catalogs: true
I20250625 13:52:07.605651 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0f82cde67a844d2dadb19c95b952db96" } modify_external_catalogs: true
W20250625 13:52:07.606174 985 leader_election.cc:343] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f065b4e3218b4e6e8a2ac3a70eede13b
I20250625 13:52:07.607573 915 raft_consensus.cc:2466] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:07.611877 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cdcf80766e3e4d9c98920cc3d711764d" } new_table_name: "default.Table_81c1b426fda24349805e3e5cf9fdd561" modify_external_catalogs: true
I20250625 13:52:07.612104 1710 tablet_replica.cc:331] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:07.612298 1710 raft_consensus.cc:2241] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.612432 1710 raft_consensus.cc:2270] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.612943 1710 ts_tablet_manager.cc:1905] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.614744 590 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c] alter complete (version 2)
I20250625 13:52:07.616746 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 29be8c0350584de382985e66556bdcca (table default.Table_13c3269d65ff41bc87efff42b2dce296 [id=1c58b83050f1494a91b0845e8dfe04bf]) successfully deleted
W20250625 13:52:07.617009 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e99fe5ad690049029b8325dd38a11580 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e99fe5ad690049029b8325dd38a11580 already in progress: creating tablet
I20250625 13:52:07.617748 983 leader_election.cc:304] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:07.617992 983 raft_consensus.cc:2684] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:52:07.620982 912 raft_consensus.cc:2466] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:07.621433 983 leader_election.cc:304] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:07.622076 1821 raft_consensus.cc:2802] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:07.622197 1821 raft_consensus.cc:695] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:07.622503 1821 consensus_queue.cc:237] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.640664 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3be2a3bc86fd4fd993dbfa89af6b7b97" } new_table_name: "default.Table_906b9e1a3a8147fb935f7a96968ed294" modify_external_catalogs: true
I20250625 13:52:07.641628 1748 raft_consensus.cc:491] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.641773 1748 raft_consensus.cc:513] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.642207 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d67dfdd23e284e7588f0d5fade2e36e7" } new_table_name: "default.Table_aed3405f8cc04d6788a5b5ba519f5363" modify_external_catalogs: true
I20250625 13:52:07.642433 1748 leader_election.cc:290] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:07.643754 849 leader_election.cc:343] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 9a31f10c8ca745b7ad635c622a265450
W20250625 13:52:07.644702 851 leader_election.cc:343] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9a31f10c8ca745b7ad635c622a265450
I20250625 13:52:07.644807 851 leader_election.cc:304] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:07.645256 1748 raft_consensus.cc:2747] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:07.647476 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a0b94879874446b569b3cb3250b185 has no consensus state
I20250625 13:52:07.649994 1165 ts_tablet_manager.cc:1918] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.650126 1165 log.cc:1199] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a0e81d4c904248eba858462bcf31c5d9
I20250625 13:52:07.650444 1165 ts_tablet_manager.cc:1939] T a0e81d4c904248eba858462bcf31c5d9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.652863 1171 tablet_bootstrap.cc:492] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.653009 1171 ts_tablet_manager.cc:1397] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.180s user 0.008s sys 0.038s
I20250625 13:52:07.653802 1171 raft_consensus.cc:357] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.654001 1171 raft_consensus.cc:383] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.654086 1171 raft_consensus.cc:738] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.654318 1171 consensus_queue.cc:260] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.654783 1171 ts_tablet_manager.cc:1428] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:07.658313 1168 ts_tablet_manager.cc:1918] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:07.658356 1710 ts_tablet_manager.cc:1918] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.658500 1168 log.cc:1199] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d0c5bea3b6214fa8af2792c5e9c411d9
I20250625 13:52:07.658497 1710 log.cc:1199] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/11711546f0e14f9a9eabf003f49a69b7
I20250625 13:52:07.658836 1710 ts_tablet_manager.cc:1939] T 11711546f0e14f9a9eabf003f49a69b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.659364 1710 tablet_replica.cc:331] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:07.659426 1165 tablet_replica.cc:331] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:07.659570 1710 raft_consensus.cc:2241] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.659605 1165 raft_consensus.cc:2241] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.659742 1165 raft_consensus.cc:2270] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.659742 1710 raft_consensus.cc:2270] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.660466 1168 ts_tablet_manager.cc:1939] T d0c5bea3b6214fa8af2792c5e9c411d9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:07.660517 1710 ts_tablet_manager.cc:1905] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.660780 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 11711546f0e14f9a9eabf003f49a69b7 (table default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0]) successfully deleted
I20250625 13:52:07.661379 1165 ts_tablet_manager.cc:1905] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.661422 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a0e81d4c904248eba858462bcf31c5d9 (table default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]) successfully deleted
I20250625 13:52:07.667129 1710 ts_tablet_manager.cc:1918] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.667232 1710 log.cc:1199] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/2cd5303e674246ada7d36574b535b333
I20250625 13:52:07.667506 1710 ts_tablet_manager.cc:1939] T 2cd5303e674246ada7d36574b535b333 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.670825 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 498) took 5910 ms. Trace:
I20250625 13:52:07.670977 1168 rpcz_store.cc:276] 0625 13:52:01.760775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.761081 (+ 306us) service_pool.cc:225] Handling call
0625 13:52:07.496065 (+5734984us) ts_tablet_manager.cc:1145] Deleting tablet d0c5bea3b6214fa8af2792c5e9c411d9
0625 13:52:07.496074 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.627017 (+130943us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.658208 (+ 31191us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.670804 (+ 12596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5734747}
I20250625 13:52:07.671460 1168 tablet_replica.cc:331] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:07.671730 1168 raft_consensus.cc:2241] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:07.671892 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 413afd7b62d24fe8b9eceadd57e29fc8 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:07.672160 1168 raft_consensus.cc:2270] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.672538 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_35b3cb8e94364083971a19c3e8bc98a9" } modify_external_catalogs: true
I20250625 13:52:07.672721 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_35b3cb8e94364083971a19c3e8bc98a9" } modify_external_catalogs: true
I20250625 13:52:07.672822 1168 ts_tablet_manager.cc:1905] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:07.673570 581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250625 13:52:07.675019 1171 tablet_bootstrap.cc:492] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:07.676656 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet d0c5bea3b6214fa8af2792c5e9c411d9 (table default.Table_77082a1ec23a48adb1f251559dc90002 [id=d482f250757e4eaaa5a5a4daa115bd34]) successfully deleted
I20250625 13:52:07.676865 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 2cd5303e674246ada7d36574b535b333 (table default.Table_09cd14059fee4f4aaadc9c4e4bd5a249 [id=a79073f0368145dcab2528663f56e9aa]) successfully deleted
W20250625 13:52:07.677083 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d2972d4366c143498f746fd62a039acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2972d4366c143498f746fd62a039acd already in progress: creating tablet
W20250625 13:52:07.677276 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ef2c3757c0284aaca70ec317c05195d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef2c3757c0284aaca70ec317c05195d6 already in progress: creating tablet
W20250625 13:52:07.677487 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e7bd4939714a4359846345ed0521701c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7bd4939714a4359846345ed0521701c already in progress: creating tablet
W20250625 13:52:07.677685 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d89b0ddcc14148faa1a6926fcbb027a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d89b0ddcc14148faa1a6926fcbb027a8 already in progress: creating tablet
W20250625 13:52:07.677920 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f065b4e3218b4e6e8a2ac3a70eede13b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f065b4e3218b4e6e8a2ac3a70eede13b already in progress: creating tablet
W20250625 13:52:07.678530 581 consensus_peers.cc:487] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 -> Peer 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015): Couldn't send request to peer 3fec9d8a79054a5991ae90d8a3101e32. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:33015: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250625 13:52:07.680835 1171 tablet_bootstrap.cc:654] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.683828 1165 ts_tablet_manager.cc:1918] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.683996 1165 log.cc:1199] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/77c50e1ebb9042ef8f0a651ba7186b74
I20250625 13:52:07.684883 1170 ts_tablet_manager.cc:1428] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.207s user 0.014s sys 0.035s
I20250625 13:52:07.686964 1165 ts_tablet_manager.cc:1939] T 77c50e1ebb9042ef8f0a651ba7186b74 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.687539 1170 tablet_bootstrap.cc:492] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.692308 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f065b4e3218b4e6e8a2ac3a70eede13b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.692713 1710 tablet_replica.cc:331] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:07.689164 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 77c50e1ebb9042ef8f0a651ba7186b74 (table default.Table_bdcf17e5612a44579ba27af546ca1def [id=cc778fd879634dd6a2ba7ba8274c9ef4]) successfully deleted
I20250625 13:52:07.692880 1710 raft_consensus.cc:2241] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:07.693102 1710 pending_rounds.cc:70] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:07.693151 1710 pending_rounds.cc:77] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520625144291328 op_type: NO_OP noop_request { }
I20250625 13:52:07.693231 1710 raft_consensus.cc:2887] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:07.693301 1710 raft_consensus.cc:2270] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:07.694099 2030 consensus_queue.cc:1162] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250625 13:52:07.696344 1165 tablet_replica.cc:331] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:07.696594 1165 raft_consensus.cc:2241] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.696766 1165 raft_consensus.cc:2270] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.695479 1170 tablet_bootstrap.cc:654] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.697149 1710 ts_tablet_manager.cc:1905] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.698215 1165 ts_tablet_manager.cc:1905] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.698706 1166 tablet_bootstrap.cc:492] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.698855 1166 ts_tablet_manager.cc:1397] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.212s user 0.010s sys 0.037s
I20250625 13:52:07.699734 1168 ts_tablet_manager.cc:1918] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:07.699707 1166 raft_consensus.cc:357] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.699899 1166 raft_consensus.cc:383] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.699965 1166 raft_consensus.cc:738] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.700212 1166 consensus_queue.cc:260] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.700215 1168 log.cc:1199] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/c911d9ecbded417daf70e85c7cbcd52e
I20250625 13:52:07.702610 1168 ts_tablet_manager.cc:1939] T c911d9ecbded417daf70e85c7cbcd52e P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:07.704926 1166 ts_tablet_manager.cc:1428] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250625 13:52:07.708402 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 499) took 5936 ms. Trace:
I20250625 13:52:07.708787 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f14db116c8184370a995e82e33813932
I20250625 13:52:07.709721 743 tablet_service.cc:1515] Processing DeleteTablet for tablet f14db116c8184370a995e82e33813932 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.708956 1168 rpcz_store.cc:276] 0625 13:52:01.771949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.772000 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:07.671417 (+5899417us) ts_tablet_manager.cc:1145] Deleting tablet c911d9ecbded417daf70e85c7cbcd52e
0625 13:52:07.671426 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.679706 (+ 8280us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.699629 (+ 19923us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.708248 (+ 8619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5898771}
I20250625 13:52:07.710263 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f14db116c8184370a995e82e33813932 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.710620 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f14db116c8184370a995e82e33813932 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.710784 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet c911d9ecbded417daf70e85c7cbcd52e (table default.Table_a90cc30efd7145fdbaa7ebdc5db04107 [id=dec0cb5442804b6398a22dbd8c9b0f3a]) successfully deleted
W20250625 13:52:07.712302 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 024f6cb363594420b0d5480fa79d156b consensus state has no leader
W20250625 13:52:07.712603 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 066db01f791542a38cfb6e408c4836ed has no consensus state
I20250625 13:52:07.714613 1166 tablet_bootstrap.cc:492] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.722491 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ab888f08b1984d8da2a92b8b396d4989"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.723083 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab888f08b1984d8da2a92b8b396d4989 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.723503 1710 ts_tablet_manager.cc:1918] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:07.723649 1710 log.cc:1199] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/f065b4e3218b4e6e8a2ac3a70eede13b
I20250625 13:52:07.723982 1166 tablet_bootstrap.cc:654] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.724032 1710 ts_tablet_manager.cc:1939] T f065b4e3218b4e6e8a2ac3a70eede13b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.725013 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet f065b4e3218b4e6e8a2ac3a70eede13b (table default.Table_71166441017840fa9a67752209689ee2 [id=fc9bcf7791ec4c069f0bcae18943cac9]) successfully deleted
I20250625 13:52:07.725050 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_237174b0502449feae068850a65184a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.725349 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f14db116c8184370a995e82e33813932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f14db116c8184370a995e82e33813932 already in progress: creating tablet
W20250625 13:52:07.725628 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_237174b0502449feae068850a65184a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.726044 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c640e0bdbd3e4274ab75511cebd6828d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.726526 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c640e0bdbd3e4274ab75511cebd6828d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.727892 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b0fa30878dd349899cc45f2600401269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.728339 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0fa30878dd349899cc45f2600401269 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.729473 1165 ts_tablet_manager.cc:1918] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.729605 1165 log.cc:1199] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d43ba17ebfec4244894d1b6e23200c33
I20250625 13:52:07.729820 1170 tablet_bootstrap.cc:492] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.729983 1170 ts_tablet_manager.cc:1397] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.003s
I20250625 13:52:07.730785 1170 raft_consensus.cc:357] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.731003 1170 raft_consensus.cc:383] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.731092 1170 raft_consensus.cc:738] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.731963 1170 consensus_queue.cc:260] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.734244 1170 ts_tablet_manager.cc:1428] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:07.734560 1170 tablet_bootstrap.cc:492] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:07.735095 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9710609da88d4b42993b3aef02416543 consensus state has no leader
I20250625 13:52:07.736601 1165 ts_tablet_manager.cc:1939] T d43ba17ebfec4244894d1b6e23200c33 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.737699 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e6dee677a6314593b3557d7e48d1504f" } new_table_name: "default.Table_30fbf695c7684ae6a586263957012855" modify_external_catalogs: true
I20250625 13:52:07.738258 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d43ba17ebfec4244894d1b6e23200c33 (table default.Table_bc6c1303fba14b97b873dd23a423bf8c [id=999e931fc6724eec9b6d8bb94bd3cf23]) successfully deleted
I20250625 13:52:07.739348 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb9a98e402b4da8a9c2bd8317cef532 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:07.739497 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d89b0ddcc14148faa1a6926fcbb027a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d89b0ddcc14148faa1a6926fcbb027a8 already in progress: creating tablet
I20250625 13:52:07.739871 1710 tablet_replica.cc:331] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:07.740054 1710 raft_consensus.cc:2241] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.740212 1710 raft_consensus.cc:2270] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:07.740190 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f14db116c8184370a995e82e33813932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f14db116c8184370a995e82e33813932 already in progress: creating tablet
I20250625 13:52:07.741073 1710 ts_tablet_manager.cc:1905] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.744750 1171 tablet_bootstrap.cc:492] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.744948 1171 ts_tablet_manager.cc:1397] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.001s
I20250625 13:52:07.745705 1171 raft_consensus.cc:357] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.745945 1171 raft_consensus.cc:383] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.746042 1171 raft_consensus.cc:738] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.746316 1171 consensus_queue.cc:260] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.749599 1171 ts_tablet_manager.cc:1428] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:07.749944 1171 tablet_bootstrap.cc:492] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:07.752765 1171 tablet_bootstrap.cc:654] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.753590 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 500) took 5735 ms. Trace:
I20250625 13:52:07.753712 1168 rpcz_store.cc:276] 0625 13:52:01.977041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.977104 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:07.711214 (+5734110us) ts_tablet_manager.cc:1145] Deleting tablet 0e3ef7f5b474466bbc19fd31c6b6d99c
0625 13:52:07.711291 (+ 77us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.712210 (+ 919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5733770}
I20250625 13:52:07.754148 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 501) took 5761 ms. Trace:
W20250625 13:52:07.754242 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e3ef7f5b474466bbc19fd31c6b6d99c already in progress: creating tablet
I20250625 13:52:07.754271 1168 rpcz_store.cc:276] 0625 13:52:01.992874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:01.992930 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:07.754026 (+5761096us) ts_tablet_manager.cc:1145] Deleting tablet 2404e712e18146b68757bd77362c1358
0625 13:52:07.754034 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.754134 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5760890}
I20250625 13:52:07.754715 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 508) took 5707 ms. Trace:
W20250625 13:52:07.754827 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2404e712e18146b68757bd77362c1358 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2404e712e18146b68757bd77362c1358 already in progress: creating tablet
I20250625 13:52:07.754846 1168 rpcz_store.cc:276] 0625 13:52:02.047148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.059362 (+ 12214us) service_pool.cc:225] Handling call
0625 13:52:07.754623 (+5695261us) ts_tablet_manager.cc:1145] Deleting tablet 48645c913fc24c03a922cf6ea7531022
0625 13:52:07.754631 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.754701 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5691166}
W20250625 13:52:07.755347 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 48645c913fc24c03a922cf6ea7531022 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48645c913fc24c03a922cf6ea7531022 already in progress: creating tablet
I20250625 13:52:07.755257 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 509) took 5707 ms. Trace:
I20250625 13:52:07.755477 1168 rpcz_store.cc:276] 0625 13:52:02.047336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.059466 (+ 12130us) service_pool.cc:225] Handling call
0625 13:52:07.755167 (+5695701us) ts_tablet_manager.cc:1145] Deleting tablet 553af7c7326240838f37e3861d4e3357
0625 13:52:07.755175 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.755243 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5691602}
I20250625 13:52:07.755964 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 510) took 5682 ms. Trace:
W20250625 13:52:07.756043 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 553af7c7326240838f37e3861d4e3357 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 553af7c7326240838f37e3861d4e3357 already in progress: creating tablet
I20250625 13:52:07.756091 1168 rpcz_store.cc:276] 0625 13:52:02.073048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.073109 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:07.755858 (+5682749us) ts_tablet_manager.cc:1145] Deleting tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57
0625 13:52:07.755866 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.755951 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5682462}
I20250625 13:52:07.756480 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 512) took 5669 ms. Trace:
W20250625 13:52:07.756567 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57 already in progress: creating tablet
I20250625 13:52:07.756675 1168 rpcz_store.cc:276] 0625 13:52:02.087096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.087191 (+ 95us) service_pool.cc:225] Handling call
0625 13:52:07.756391 (+5669200us) ts_tablet_manager.cc:1145] Deleting tablet 9c1deb52366c404cbdc08076d4bcb3d1
0625 13:52:07.756400 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.756466 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5668974}
I20250625 13:52:07.757030 1168 tablet_replica.cc:331] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:07.757186 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9c1deb52366c404cbdc08076d4bcb3d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c1deb52366c404cbdc08076d4bcb3d1 already in progress: creating tablet
I20250625 13:52:07.757326 1168 raft_consensus.cc:2241] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:07.757676 1168 raft_consensus.cc:2270] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.758226 1168 ts_tablet_manager.cc:1905] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.763355 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_42a2e34e5d204e7188555f5e5616cc4c" } modify_external_catalogs: true
I20250625 13:52:07.763614 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_42a2e34e5d204e7188555f5e5616cc4c" } modify_external_catalogs: true
I20250625 13:52:07.763836 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e2e216856d5a4a3f8cd3a3852ffe2c31
I20250625 13:52:07.763875 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f7f814de767d41a4b070ac5045680405"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:07.765229 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.765800 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.766049 889 tablet_service.cc:1515] Processing DeleteTablet for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:07.766889 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7f814de767d41a4b070ac5045680405 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:07.767066 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2e216856d5a4a3f8cd3a3852ffe2c31 already in progress: creating tablet
I20250625 13:52:07.768190 595 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_107a64d05b814c5a9cc7b956f38af315 [id=6bfad46432b7473f88d77bfcfa2a4e9a] alter complete (version 2)
I20250625 13:52:07.768723 889 tablet_service.cc:1515] Processing DeleteTablet for tablet c911d9ecbded417daf70e85c7cbcd52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:54 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.772768 1170 tablet_bootstrap.cc:654] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.780385 1710 ts_tablet_manager.cc:1918] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.780588 1710 log.cc:1199] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ecb9a98e402b4da8a9c2bd8317cef532
I20250625 13:52:07.780980 1710 ts_tablet_manager.cc:1939] T ecb9a98e402b4da8a9c2bd8317cef532 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.781651 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ecb9a98e402b4da8a9c2bd8317cef532 (table default.Table_cc7e074a01254cbb9272fd4ea7dba124 [id=d85d6e75f7e9420096456eac1ab144cc]) successfully deleted
W20250625 13:52:07.782105 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2e216856d5a4a3f8cd3a3852ffe2c31 already in progress: creating tablet
I20250625 13:52:07.783193 743 tablet_service.cc:1515] Processing DeleteTablet for tablet b14d00c93a964edd815122e3723f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.783506 1165 tablet_replica.cc:331] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:07.783778 1165 raft_consensus.cc:2241] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.783982 1165 raft_consensus.cc:2270] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.784426 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 171ce24a435e40e48026039941ca7883 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:07.785050 1165 ts_tablet_manager.cc:1905] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.785120 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a47b9f65587049a1901b3490529d730f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:07.785671 1170 tablet_bootstrap.cc:492] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:07.785703 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a47b9f65587049a1901b3490529d730f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.785822 1170 ts_tablet_manager.cc:1397] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.005s
I20250625 13:52:07.786523 1170 raft_consensus.cc:357] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.787581 1170 raft_consensus.cc:383] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.787659 1170 raft_consensus.cc:738] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.787914 1170 consensus_queue.cc:260] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.789888 1170 ts_tablet_manager.cc:1428] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250625 13:52:07.790172 1170 tablet_bootstrap.cc:492] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.791395 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bbd509e3dfb45d98314c6afe37677f6 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bd51cad7d6342d7a1d45f59653fa32f) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.792457 1170 tablet_bootstrap.cc:654] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.796684 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1dc56df21cb14dd1b0c006c14bf997e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.797320 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dc56df21cb14dd1b0c006c14bf997e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.797791 1165 ts_tablet_manager.cc:1918] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.797933 1165 log.cc:1199] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/b14d00c93a964edd815122e3723f89e7
I20250625 13:52:07.798318 1165 ts_tablet_manager.cc:1939] T b14d00c93a964edd815122e3723f89e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.799465 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet b14d00c93a964edd815122e3723f89e7 (table default.Table_295dd4f121234e659090b8e4cf70faee [id=47cd45886fc94cd7ba5cafa60838d601]) successfully deleted
I20250625 13:52:07.799763 1166 tablet_bootstrap.cc:492] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.799945 1166 ts_tablet_manager.cc:1397] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.000s
I20250625 13:52:07.800956 1166 raft_consensus.cc:357] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.801187 1166 raft_consensus.cc:383] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.801265 1166 raft_consensus.cc:738] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.801524 1166 consensus_queue.cc:260] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.802158 1166 ts_tablet_manager.cc:1428] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:07.803743 1710 tablet_replica.cc:331] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:07.804030 1710 raft_consensus.cc:2241] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.805831 1710 raft_consensus.cc:2270] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:07.809262 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af704cdf44ef42f7b5ea10b199d0f017 consensus state has no leader
I20250625 13:52:07.813706 1842 raft_consensus.cc:491] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.813860 1842 raft_consensus.cc:513] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.814495 1842 leader_election.cc:290] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:07.815114 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9267df231f354c578c38f452418e523c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:07.815685 849 leader_election.cc:343] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.816314 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9267df231f354c578c38f452418e523c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:07.816521 777 raft_consensus.cc:2466] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:07.816934 851 leader_election.cc:304] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:07.817308 1842 raft_consensus.cc:2802] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:07.817348 1171 tablet_bootstrap.cc:492] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.817379 1842 raft_consensus.cc:491] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:07.817438 1842 raft_consensus.cc:3058] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:07.817462 1171 ts_tablet_manager.cc:1397] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.068s user 0.000s sys 0.003s
I20250625 13:52:07.818173 1171 raft_consensus.cc:357] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.818337 1171 raft_consensus.cc:383] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.818385 1171 raft_consensus.cc:738] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.819761 1171 consensus_queue.cc:260] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.820096 1842 raft_consensus.cc:513] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.820698 1842 leader_election.cc:290] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:07.821074 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9267df231f354c578c38f452418e523c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:07.821164 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 067272869c5547648c90f83a34a036aa with delete_type TABLET_DATA_DELETED (replaced by tablet 6340de862fd04c12bb7f85557cfe53d2) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.821422 1165 tablet_replica.cc:331] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:07.821640 1165 raft_consensus.cc:2241] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:07.821681 849 leader_election.cc:343] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.821792 1165 raft_consensus.cc:2270] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.822319 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9267df231f354c578c38f452418e523c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:07.822449 1710 ts_tablet_manager.cc:1905] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.822464 777 raft_consensus.cc:3058] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:07.824983 1165 ts_tablet_manager.cc:1905] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.825088 777 raft_consensus.cc:2466] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:07.827493 851 leader_election.cc:304] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:07.827823 1170 tablet_bootstrap.cc:492] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.827972 1170 ts_tablet_manager.cc:1397] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.006s
I20250625 13:52:07.828017 1837 raft_consensus.cc:2802] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:07.828136 1837 raft_consensus.cc:695] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:07.828362 1837 consensus_queue.cc:237] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.829270 1170 raft_consensus.cc:357] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.829457 1170 raft_consensus.cc:383] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.829538 1170 raft_consensus.cc:738] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.829769 1170 consensus_queue.cc:260] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.831652 1166 tablet_bootstrap.cc:492] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.833068 1170 ts_tablet_manager.cc:1428] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250625 13:52:07.833201 1166 tablet_bootstrap.cc:654] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.833762 1170 tablet_bootstrap.cc:492] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.834877 1710 ts_tablet_manager.cc:1918] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.835016 1710 log.cc:1199] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/2bbd509e3dfb45d98314c6afe37677f6
I20250625 13:52:07.835939 1710 ts_tablet_manager.cc:1939] T 2bbd509e3dfb45d98314c6afe37677f6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.836583 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet fd39216b01054ff38ec9175d7a2f87f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.838481 1710 tablet_replica.cc:331] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:07.838594 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 2bbd509e3dfb45d98314c6afe37677f6 (table default.Table_683f20e141ca4122bee28d4e86aece20 [id=40f54158a1c84f5988762e7a44e3eab1]) successfully deleted
I20250625 13:52:07.838702 1710 raft_consensus.cc:2241] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.838863 1710 raft_consensus.cc:2270] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.843715 1170 tablet_bootstrap.cc:654] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.845844 1165 ts_tablet_manager.cc:1918] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.844513 1710 ts_tablet_manager.cc:1905] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.846798 1165 log.cc:1199] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/067272869c5547648c90f83a34a036aa
I20250625 13:52:07.847158 1165 ts_tablet_manager.cc:1939] T 067272869c5547648c90f83a34a036aa P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.846694 1166 tablet_bootstrap.cc:492] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.848110 1166 ts_tablet_manager.cc:1397] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20250625 13:52:07.848953 1166 raft_consensus.cc:357] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.850203 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 067272869c5547648c90f83a34a036aa (table default.Table_baa360c7bf404bb79b9e32c5d95dc8b5 [id=a1f1874f2c0d454694a40c5d5fcf4485]) successfully deleted
I20250625 13:52:07.850625 1166 raft_consensus.cc:383] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.850852 1166 raft_consensus.cc:738] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.851207 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_56560af1ef3144d4a8daed436f4b0505" } new_table_name: "default.Table_4c2981fcc6824fd4aa3f0c4eb5b82d6c" modify_external_catalogs: true
I20250625 13:52:07.851344 1166 consensus_queue.cc:260] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.853912 1171 ts_tablet_manager.cc:1428] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20250625 13:52:07.858680 743 tablet_service.cc:1515] Processing DeleteTablet for tablet df1ed922750f45eab02d1fc0c26af75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.860312 1046 raft_consensus.cc:3058] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:07.864758 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet df1ed922750f45eab02d1fc0c26af75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df1ed922750f45eab02d1fc0c26af75e already in progress: creating tablet
I20250625 13:52:07.867345 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a5675aff497c4f018385318a042343f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.867957 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5675aff497c4f018385318a042343f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.870054 1166 ts_tablet_manager.cc:1428] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20250625 13:52:07.872845 1710 ts_tablet_manager.cc:1918] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.873446 1710 log.cc:1199] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/fd39216b01054ff38ec9175d7a2f87f2
I20250625 13:52:07.873868 1710 ts_tablet_manager.cc:1939] T fd39216b01054ff38ec9175d7a2f87f2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:07.875192 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet fd39216b01054ff38ec9175d7a2f87f2 (table default.Table_c54f67fabc114fc6bb555987a103adba [id=aa481ec0dc6e42b391641f4c3d05ca42]) successfully deleted
I20250625 13:52:07.875726 1171 tablet_bootstrap.cc:492] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:07.876004 1045 raft_consensus.cc:3058] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:07.877978 1166 tablet_bootstrap.cc:492] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.881245 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_509942f9ab5846c88eabce5d076d1c35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.882004 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_509942f9ab5846c88eabce5d076d1c35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.882114 743 tablet_service.cc:1515] Processing DeleteTablet for tablet df1ed922750f45eab02d1fc0c26af75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.882339 1165 tablet_replica.cc:331] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:07.882509 1165 raft_consensus.cc:2241] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.882678 1165 raft_consensus.cc:2270] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.883515 1165 ts_tablet_manager.cc:1905] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:07.884122 1166 tablet_bootstrap.cc:654] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.885198 1171 tablet_bootstrap.cc:654] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.890059 1165 ts_tablet_manager.cc:1918] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:07.890204 1165 log.cc:1199] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/df1ed922750f45eab02d1fc0c26af75e
I20250625 13:52:07.890488 1165 ts_tablet_manager.cc:1939] T df1ed922750f45eab02d1fc0c26af75e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:07.865669 1168 ts_tablet_manager.cc:1918] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250625 13:52:07.891996 1168 log.cc:1199] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/6e0f2fa5d5614ad394e6e12ec62814d4
I20250625 13:52:07.903574 1168 ts_tablet_manager.cc:1939] T 6e0f2fa5d5614ad394e6e12ec62814d4 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:07.904510 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 534) took 5571 ms. Trace:
I20250625 13:52:07.904611 1168 rpcz_store.cc:276] 0625 13:52:02.336052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.336113 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:07.756992 (+5420879us) ts_tablet_manager.cc:1145] Deleting tablet 6e0f2fa5d5614ad394e6e12ec62814d4
0625 13:52:07.757000 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.847878 (+ 90878us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.865577 (+ 17699us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:07.904485 (+ 38908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5390770}
I20250625 13:52:07.904793 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet df1ed922750f45eab02d1fc0c26af75e (table default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]) successfully deleted
I20250625 13:52:07.905040 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 519) took 5604 ms. Trace:
I20250625 13:52:07.905128 1168 rpcz_store.cc:276] 0625 13:52:02.300301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.310488 (+ 10187us) service_pool.cc:225] Handling call
0625 13:52:07.904888 (+5594400us) ts_tablet_manager.cc:1145] Deleting tablet fb4973cbffc743dba98d7b7c9f76be3f
0625 13:52:07.904896 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.904994 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5472206}
I20250625 13:52:07.905216 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 6e0f2fa5d5614ad394e6e12ec62814d4 (table default.Table_1ffaf5826eda45159d7cef2f2505383a [id=a64872263d0d4aeaaac5a6df29cb73bd]) successfully deleted
I20250625 13:52:07.905510 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 537) took 5560 ms. Trace:
I20250625 13:52:07.905601 1168 rpcz_store.cc:276] 0625 13:52:02.344789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.432727 (+ 87938us) service_pool.cc:225] Handling call
0625 13:52:07.905373 (+5472646us) ts_tablet_manager.cc:1145] Deleting tablet 8063122ea7b24b6790f5c3cc884e09be
0625 13:52:07.905381 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.905468 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5437676}
I20250625 13:52:07.905567 2050 tablet.cc:1722] T 88f72c555d9842f28ca4bf310da9c53c P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:07.905225 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b14d3c37ddaa46468b60d59d70b9ccba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:07.905911 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 535) took 5569 ms. Trace:
I20250625 13:52:07.905983 1168 rpcz_store.cc:276] 0625 13:52:02.336364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.336416 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:07.905805 (+5569389us) ts_tablet_manager.cc:1145] Deleting tablet 69672579a83c41d09ee7c78a6fa7d746
0625 13:52:07.905813 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.905879 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5424298}
W20250625 13:52:07.906052 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f47260bf0ad41fdbfbad50380a12547 consensus state has no leader
I20250625 13:52:07.906268 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 538) took 5545 ms. Trace:
W20250625 13:52:07.906298 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b14d3c37ddaa46468b60d59d70b9ccba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.906352 1168 rpcz_store.cc:276] 0625 13:52:02.361093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.467721 (+106628us) service_pool.cc:225] Handling call
0625 13:52:07.906156 (+5438435us) ts_tablet_manager.cc:1145] Deleting tablet 5d4fa8abea8d4eee8651be9bf4b2b34f
0625 13:52:07.906163 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.906238 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5326750}
I20250625 13:52:07.906661 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 539) took 5329 ms. Trace:
I20250625 13:52:07.906736 1168 rpcz_store.cc:276] 0625 13:52:02.577516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.577708 (+ 192us) service_pool.cc:225] Handling call
0625 13:52:07.906536 (+5328828us) ts_tablet_manager.cc:1145] Deleting tablet a6ff3b675aba42adb035ff5d1404a889
0625 13:52:07.906563 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.906626 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5324072}
I20250625 13:52:07.907071 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 541) took 5268 ms. Trace:
W20250625 13:52:07.907507 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet fb4973cbffc743dba98d7b7c9f76be3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb4973cbffc743dba98d7b7c9f76be3f already in progress: creating tablet
W20250625 13:52:07.907792 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8063122ea7b24b6790f5c3cc884e09be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8063122ea7b24b6790f5c3cc884e09be already in progress: creating tablet
W20250625 13:52:07.908005 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 69672579a83c41d09ee7c78a6fa7d746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69672579a83c41d09ee7c78a6fa7d746 already in progress: creating tablet
W20250625 13:52:07.908205 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 5d4fa8abea8d4eee8651be9bf4b2b34f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d4fa8abea8d4eee8651be9bf4b2b34f already in progress: creating tablet
W20250625 13:52:07.908427 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a6ff3b675aba42adb035ff5d1404a889 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6ff3b675aba42adb035ff5d1404a889 already in progress: creating tablet
I20250625 13:52:07.909782 590 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 88f72c555d9842f28ca4bf310da9c53c (table default.Table_8013d5185a734552a6261f57235f3689 [id=97da86ea5100489ca9e82144e359d63c]). Expected version 2 got 0
I20250625 13:52:07.910995 1171 tablet_bootstrap.cc:492] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.911592 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e23d05f0e5074305a2cdbd567759c6d3" } modify_external_catalogs: true
I20250625 13:52:07.911784 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e23d05f0e5074305a2cdbd567759c6d3" } modify_external_catalogs: true
I20250625 13:52:07.912488 1171 ts_tablet_manager.cc:1397] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20250625 13:52:07.914546 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_da4c3565bbf845ca9a26fe3d1f39f99f" } new_table_name: "default.Table_d8668c84f52c4e7299ffffb2c41a9929" modify_external_catalogs: true
I20250625 13:52:07.916778 1171 raft_consensus.cc:357] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.916960 1171 raft_consensus.cc:383] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.917034 1171 raft_consensus.cc:738] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.917292 1171 consensus_queue.cc:260] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.907162 1168 rpcz_store.cc:276] 0625 13:52:02.638940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.639006 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:07.906961 (+5267955us) ts_tablet_manager.cc:1145] Deleting tablet 2c8cf5d6cd624a1797486c69bd2098fd
0625 13:52:07.906968 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.907045 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5266952}
W20250625 13:52:07.917750 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2c8cf5d6cd624a1797486c69bd2098fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c8cf5d6cd624a1797486c69bd2098fd already in progress: creating tablet
I20250625 13:52:07.918133 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 542) took 5277 ms. Trace:
I20250625 13:52:07.918231 1168 rpcz_store.cc:276] 0625 13:52:02.640319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.640375 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:07.917979 (+5277604us) ts_tablet_manager.cc:1145] Deleting tablet 57afeff143c846289c6159974b00d7ba
0625 13:52:07.917989 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.918077 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5275154}
I20250625 13:52:07.918550 1170 tablet_bootstrap.cc:492] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:07.918618 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 57afeff143c846289c6159974b00d7ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57afeff143c846289c6159974b00d7ba already in progress: creating tablet
I20250625 13:52:07.918821 1170 ts_tablet_manager.cc:1397] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.012s
I20250625 13:52:07.919974 1170 raft_consensus.cc:357] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.920217 1170 raft_consensus.cc:383] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.920346 1170 raft_consensus.cc:738] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.920678 1170 consensus_queue.cc:260] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.921902 1170 ts_tablet_manager.cc:1428] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250625 13:52:07.922318 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81a359c798024b2a99aa3b307cc1967f consensus state has no leader
I20250625 13:52:07.922691 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a382ff2909274bbd88d364ade2fe1080" } new_table_name: "default.Table_3ab9801efffc44f183b50b4081652e9f" modify_external_catalogs: true
I20250625 13:52:07.923686 1170 tablet_bootstrap.cc:492] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.926975 1170 tablet_bootstrap.cc:654] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.928270 1171 ts_tablet_manager.cc:1428] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 13:52:07.928689 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 543) took 5281 ms. Trace:
W20250625 13:52:07.933902 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0012e95ff4924723a6e6aaf6b85da438 consensus state has no leader
I20250625 13:52:07.937312 1168 rpcz_store.cc:276] 0625 13:52:02.647010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.647068 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:07.928507 (+5281439us) ts_tablet_manager.cc:1145] Deleting tablet 39540b81fd4746858e65be49ded0e1df
0625 13:52:07.928530 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.928638 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5277794}
I20250625 13:52:07.937661 1171 tablet_bootstrap.cc:492] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:07.938601 1166 tablet_bootstrap.cc:492] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:07.938621 1170 tablet_bootstrap.cc:492] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.938774 1170 ts_tablet_manager.cc:1397] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.001s
I20250625 13:52:07.938813 1166 ts_tablet_manager.cc:1397] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.006s
I20250625 13:52:07.941061 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_914aeef494b1417b96fd83b69c631c69" } new_table_name: "default.Table_f3b3b6035e8e4cf9872da9d67e09e7ae" modify_external_catalogs: true
I20250625 13:52:07.941819 1166 raft_consensus.cc:357] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.942936 1170 raft_consensus.cc:357] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.943178 1170 raft_consensus.cc:383] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.943288 1170 raft_consensus.cc:738] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.943665 1170 consensus_queue.cc:260] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.945969 1170 ts_tablet_manager.cc:1428] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 13:52:07.946979 1170 tablet_bootstrap.cc:492] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.948071 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 171ce24a435e40e48026039941ca7883
I20250625 13:52:07.948459 593 catalog_manager.cc:6652] ReplaceTablet: tablet 171ce24a435e40e48026039941ca7883 deleted and replaced by tablet 2770194711ec4a86bb1354517b095f60
I20250625 13:52:07.949259 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 171ce24a435e40e48026039941ca7883 with delete_type TABLET_DATA_DELETED (replaced by tablet 2770194711ec4a86bb1354517b095f60) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.949841 1170 tablet_bootstrap.cc:654] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.949852 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 171ce24a435e40e48026039941ca7883 with delete_type TABLET_DATA_DELETED (replaced by tablet 2770194711ec4a86bb1354517b095f60) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:07.949985 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 171ce24a435e40e48026039941ca7883 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 171ce24a435e40e48026039941ca7883 already in progress: creating tablet
W20250625 13:52:07.951045 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cc353b88a124fac801b8898b5a5f72c consensus state has no leader
W20250625 13:52:07.951169 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 171ce24a435e40e48026039941ca7883 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 171ce24a435e40e48026039941ca7883 already in progress: creating tablet
I20250625 13:52:07.941983 1166 raft_consensus.cc:383] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.952627 1166 raft_consensus.cc:738] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:07.952884 1166 consensus_queue.cc:260] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.958827 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 47eb205a9bcd4608930d9355978a44ce
I20250625 13:52:07.959124 1170 tablet_bootstrap.cc:492] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:07.959342 1170 ts_tablet_manager.cc:1397] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250625 13:52:07.959983 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 47eb205a9bcd4608930d9355978a44ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.959978 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 47eb205a9bcd4608930d9355978a44ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.960237 1170 raft_consensus.cc:357] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.960431 1170 raft_consensus.cc:383] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:07.960537 1170 raft_consensus.cc:738] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:07.960780 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 47eb205a9bcd4608930d9355978a44ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47eb205a9bcd4608930d9355978a44ce already in progress: creating tablet
W20250625 13:52:07.960803 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 47eb205a9bcd4608930d9355978a44ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47eb205a9bcd4608930d9355978a44ce already in progress: creating tablet
I20250625 13:52:07.960763 1170 consensus_queue.cc:260] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:07.960968 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
I20250625 13:52:07.962253 1170 ts_tablet_manager.cc:1428] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 13:52:07.962551 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 664a83b20b344584b5c85db53bfa9021
I20250625 13:52:07.962997 1166 ts_tablet_manager.cc:1428] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20250625 13:52:07.963294 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 664a83b20b344584b5c85db53bfa9021 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:07.964458 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 664a83b20b344584b5c85db53bfa9021 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:07.964459 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 664a83b20b344584b5c85db53bfa9021 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 664a83b20b344584b5c85db53bfa9021 already in progress: creating tablet
I20250625 13:52:07.964656 1170 tablet_bootstrap.cc:492] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:07.966951 1170 tablet_bootstrap.cc:654] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:07.966954 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 664a83b20b344584b5c85db53bfa9021 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 664a83b20b344584b5c85db53bfa9021 already in progress: creating tablet
I20250625 13:52:07.969152 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
I20250625 13:52:07.970042 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6b6fcfe7007d45478310c91778e8fdff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.970573 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b6fcfe7007d45478310c91778e8fdff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.970966 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 944759869ced45f09dfad8959b32edff
I20250625 13:52:07.971848 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 944759869ced45f09dfad8959b32edff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:07.972465 1046 raft_consensus.cc:3058] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:07.972536 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 944759869ced45f09dfad8959b32edff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:07.972865 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 944759869ced45f09dfad8959b32edff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 944759869ced45f09dfad8959b32edff already in progress: creating tablet
I20250625 13:52:07.973927 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2a8634c60e08492faffd30a254d0dc11" } modify_external_catalogs: true
I20250625 13:52:07.973088 1166 tablet_bootstrap.cc:492] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:07.974124 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2a8634c60e08492faffd30a254d0dc11" } modify_external_catalogs: true
I20250625 13:52:07.974346 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_50aa6fee302644afa6b847fd604555b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.974853 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50aa6fee302644afa6b847fd604555b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.974130 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8cae88538eb141a281ef6a3405a519aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:07.975852 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cae88538eb141a281ef6a3405a519aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.942575 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.942737 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 36b151e200e1430c99ac194bc55881c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.948884 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 171ce24a435e40e48026039941ca7883 with delete_type TABLET_DATA_DELETED (replaced by tablet 2770194711ec4a86bb1354517b095f60) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.954463 1837 raft_consensus.cc:491] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.976547 1837 raft_consensus.cc:513] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:07.977126 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a55e69e3ed874f37bfe161cf46b4dfb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:07.977269 1837 leader_election.cc:290] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:07.957221 1842 raft_consensus.cc:491] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:07.977465 1842 raft_consensus.cc:513] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:07.977631 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a55e69e3ed874f37bfe161cf46b4dfb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:07.977862 1171 tablet_bootstrap.cc:654] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.978025 1842 leader_election.cc:290] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:07.978869 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e77736191544e8da32ca9927f0a5804" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:07.978936 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 544) took 5304 ms. Trace:
I20250625 13:52:07.960197 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 47eb205a9bcd4608930d9355978a44ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.964090 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 664a83b20b344584b5c85db53bfa9021 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.979020 1168 rpcz_store.cc:276] 0625 13:52:02.674838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:02.676091 (+ 1253us) service_pool.cc:225] Handling call
0625 13:52:07.978758 (+5302667us) ts_tablet_manager.cc:1145] Deleting tablet 60ef50a3624b4c79acc803ff0e2ed364
0625 13:52:07.978769 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.978886 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5287490}
I20250625 13:52:07.972481 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 944759869ced45f09dfad8959b32edff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:07.979562 849 leader_election.cc:343] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 19017cc372af43b2b5074a8990375ede
W20250625 13:52:07.979732 849 leader_election.cc:343] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:07.980185 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a2acecefed724eb3b55116b8e5452853
W20250625 13:52:07.980540 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 39540b81fd4746858e65be49ded0e1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39540b81fd4746858e65be49ded0e1df already in progress: creating tablet
W20250625 13:52:07.980822 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 60ef50a3624b4c79acc803ff0e2ed364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ef50a3624b4c79acc803ff0e2ed364 already in progress: creating tablet
I20250625 13:52:07.981671 891 tablet_service.cc:1515] Processing DeleteTablet for tablet a2acecefed724eb3b55116b8e5452853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:07.982046 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a2acecefed724eb3b55116b8e5452853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:07.982057 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 944759869ced45f09dfad8959b32edff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 944759869ced45f09dfad8959b32edff already in progress: creating tablet
I20250625 13:52:07.982228 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 575) took 4940 ms. Trace:
I20250625 13:52:07.982311 1168 rpcz_store.cc:276] 0625 13:52:03.042157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.069032 (+ 26875us) service_pool.cc:225] Handling call
0625 13:52:07.979272 (+4910240us) ts_tablet_manager.cc:1145] Deleting tablet d2f545a5934748b2af767d37ea937fca
0625 13:52:07.979283 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.982157 (+ 2874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4900293}
I20250625 13:52:07.982657 1168 tablet_replica.cc:331] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:07.983197 743 tablet_service.cc:1515] Processing DeleteTablet for tablet a2acecefed724eb3b55116b8e5452853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:07.983492 851 leader_election.cc:343] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 0e77736191544e8da32ca9927f0a5804
I20250625 13:52:07.983603 851 leader_election.cc:304] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:07.984025 851 leader_election.cc:343] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 19017cc372af43b2b5074a8990375ede
I20250625 13:52:07.984130 851 leader_election.cc:304] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:07.984711 1842 raft_consensus.cc:2747] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:07.984762 1168 raft_consensus.cc:2241] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:07.984908 1842 raft_consensus.cc:2747] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:07.984947 1168 raft_consensus.cc:2270] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:07.985540 1168 ts_tablet_manager.cc:1905] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:07.985551 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a2acecefed724eb3b55116b8e5452853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2acecefed724eb3b55116b8e5452853 already in progress: creating tablet
W20250625 13:52:07.985765 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d2f545a5934748b2af767d37ea937fca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2f545a5934748b2af767d37ea937fca already in progress: creating tablet
I20250625 13:52:07.987339 1166 tablet_bootstrap.cc:654] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:07.994498 2050 tablet.cc:1722] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 13:52:07.995776 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a2acecefed724eb3b55116b8e5452853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2acecefed724eb3b55116b8e5452853 already in progress: creating tablet
I20250625 13:52:07.998615 1171 tablet_bootstrap.cc:492] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:07.998854 1171 ts_tablet_manager.cc:1397] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.002s
I20250625 13:52:08.001801 1170 tablet_bootstrap.cc:492] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.001916 1171 raft_consensus.cc:357] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.002053 1170 ts_tablet_manager.cc:1397] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.008s
I20250625 13:52:08.002775 1171 raft_consensus.cc:383] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.002840 1170 raft_consensus.cc:357] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.002934 1171 raft_consensus.cc:738] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.003108 1170 raft_consensus.cc:383] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.003242 1170 raft_consensus.cc:738] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.003223 1171 consensus_queue.cc:260] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.003639 1170 consensus_queue.cc:260] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.005049 1170 ts_tablet_manager.cc:1428] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:08.005414 1170 tablet_bootstrap.cc:492] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.007277 1170 tablet_bootstrap.cc:654] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.008917 1171 ts_tablet_manager.cc:1428] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 13:52:08.013978 1171 tablet_bootstrap.cc:492] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.015717 1171 tablet_bootstrap.cc:654] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.017130 1170 tablet_bootstrap.cc:492] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.017340 1170 ts_tablet_manager.cc:1397] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20250625 13:52:08.018075 1170 raft_consensus.cc:357] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.018308 1170 raft_consensus.cc:383] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.018412 1170 raft_consensus.cc:738] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.018697 1170 consensus_queue.cc:260] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.019963 1170 ts_tablet_manager.cc:1428] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:08.020695 1168 ts_tablet_manager.cc:1918] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.020812 1168 log.cc:1199] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/04479dfb34584f0891c604ef72bf93b5
I20250625 13:52:08.020931 1170 tablet_bootstrap.cc:492] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.021095 1168 ts_tablet_manager.cc:1939] T 04479dfb34584f0891c604ef72bf93b5 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.021303 1837 raft_consensus.cc:491] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.021421 1837 raft_consensus.cc:513] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.022037 1837 leader_election.cc:290] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.022512 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36b151e200e1430c99ac194bc55881c1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.022694 1046 raft_consensus.cc:2466] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:08.023036 851 leader_election.cc:343] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 36b151e200e1430c99ac194bc55881c1
I20250625 13:52:08.023192 849 leader_election.cc:304] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.024010 1170 tablet_bootstrap.cc:654] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.025434 1171 tablet_bootstrap.cc:492] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.025561 1171 ts_tablet_manager.cc:1397] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250625 13:52:08.028537 1837 raft_consensus.cc:2802] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.028642 1837 raft_consensus.cc:491] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.028712 1837 raft_consensus.cc:3058] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.028617 1171 raft_consensus.cc:357] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.028913 1171 raft_consensus.cc:383] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.028963 1171 raft_consensus.cc:738] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.029165 1171 consensus_queue.cc:260] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.029237 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet dc75a0a8421b457089cc707d9eb6b76f (table default.Table_f618b45e019044ea946538ed3fa05471 [id=fa0246aff3f648538e6fddcb30f3a399]). Expected version 3 got 0
I20250625 13:52:08.029604 1171 ts_tablet_manager.cc:1428] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:08.029641 591 catalog_manager.cc:5582] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:08.030042 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 576) took 4714 ms. Trace:
I20250625 13:52:08.030175 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:08.030123 1168 rpcz_store.cc:276] 0625 13:52:03.315257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.316917 (+ 1660us) service_pool.cc:225] Handling call
0625 13:52:07.982569 (+4665652us) ts_tablet_manager.cc:1145] Deleting tablet 04479dfb34584f0891c604ef72bf93b5
0625 13:52:07.982578 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:07.997215 (+ 14637us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.020588 (+ 23373us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.030023 (+ 9435us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3312,"tablet-delete.queue_time_us":4622633}
I20250625 13:52:08.030503 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 593) took 4569 ms. Trace:
I20250625 13:52:08.030576 1168 rpcz_store.cc:276] 0625 13:52:03.460665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.460861 (+ 196us) service_pool.cc:225] Handling call
0625 13:52:08.030388 (+4569527us) ts_tablet_manager.cc:1145] Deleting tablet c0f461a7a7ef45009279716a3024ee26
0625 13:52:08.030398 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.030488 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4564537}
I20250625 13:52:08.032460 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 04479dfb34584f0891c604ef72bf93b5 (table default.Table_c1e172984ca643a9b7c88d09f4d5221a [id=d0ac91ff8e7f473b8b2f938243067eac]) successfully deleted
W20250625 13:52:08.032791 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c0f461a7a7ef45009279716a3024ee26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f461a7a7ef45009279716a3024ee26 already in progress: creating tablet
I20250625 13:52:08.035794 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 7d7b671a935b43a6a80a37d38e57f230 (DEFAULT_TABLE table=default.Table_5a6a094978314b53847a823ba43ac189 [id=dae6ef5d6bc5468da5b32bf934c386f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.036345 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7b671a935b43a6a80a37d38e57f230. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.030836 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 592) took 4570 ms. Trace:
I20250625 13:52:08.036999 1842 raft_consensus.cc:491] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.042491 1842 raft_consensus.cc:513] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.043233 1842 leader_election.cc:290] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.033042 1837 raft_consensus.cc:513] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:08.044201 849 leader_election.cc:343] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 46316eb725bc47e695af476c0b185504
W20250625 13:52:08.044325 851 leader_election.cc:343] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 46316eb725bc47e695af476c0b185504
I20250625 13:52:08.044445 851 leader_election.cc:304] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.044741 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36b151e200e1430c99ac194bc55881c1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:08.044915 1046 raft_consensus.cc:3058] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:08.045396 851 leader_election.cc:343] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 36b151e200e1430c99ac194bc55881c1
I20250625 13:52:08.045559 1842 raft_consensus.cc:2747] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.044346 1837 leader_election.cc:290] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.040915 1171 tablet_bootstrap.cc:492] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.047106 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_98287f576a2f4435a569d69b7abf05f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.042443 1168 rpcz_store.cc:276] 0625 13:52:03.460577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.460840 (+ 263us) service_pool.cc:225] Handling call
0625 13:52:08.030754 (+4569914us) ts_tablet_manager.cc:1145] Deleting tablet f197ea67c34546a7981bcd9df94702e6
0625 13:52:08.030762 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.030824 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4564503}
I20250625 13:52:08.047456 1168 tablet_replica.cc:331] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.047503 1046 raft_consensus.cc:2466] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.047657 1168 raft_consensus.cc:2241] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:08.047642 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet f197ea67c34546a7981bcd9df94702e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f197ea67c34546a7981bcd9df94702e6 already in progress: creating tablet
W20250625 13:52:08.047762 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98287f576a2f4435a569d69b7abf05f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.047808 1168 raft_consensus.cc:2270] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.048276 849 leader_election.cc:304] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.048517 1837 raft_consensus.cc:2802] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.048624 1837 raft_consensus.cc:695] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:08.048828 1837 consensus_queue.cc:237] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.049160 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_74ae228ed14a49bc95616379e50d5936"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.048310 1168 ts_tablet_manager.cc:1905] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.051510 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6180acf99695440fbee78730755721b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.051255 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_50bfa9943b744384b9b3911caaf604b3" } new_table_name: "default.Table_77e9a4274db94e1aac527ee57c527ade" modify_external_catalogs: true
W20250625 13:52:08.053345 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6180acf99695440fbee78730755721b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:08.054069 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ae228ed14a49bc95616379e50d5936 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.057032 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 490c8902efeb46c99543747a8787c1c7
I20250625 13:52:08.057768 1168 ts_tablet_manager.cc:1918] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.057886 1168 log.cc:1199] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/f033bfa4925b4d1c971789eb77efd92c
I20250625 13:52:08.058182 1168 ts_tablet_manager.cc:1939] T f033bfa4925b4d1c971789eb77efd92c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.058168 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 490c8902efeb46c99543747a8787c1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.058530 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 490c8902efeb46c99543747a8787c1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.058909 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 578) took 4601 ms. Trace:
I20250625 13:52:08.058990 1168 rpcz_store.cc:276] 0625 13:52:03.457928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.458273 (+ 345us) service_pool.cc:225] Handling call
0625 13:52:08.047408 (+4589135us) ts_tablet_manager.cc:1145] Deleting tablet f033bfa4925b4d1c971789eb77efd92c
0625 13:52:08.047418 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.054807 (+ 7389us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.057673 (+ 2866us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.058891 (+ 1218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4581034}
W20250625 13:52:08.059324 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 490c8902efeb46c99543747a8787c1c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 490c8902efeb46c99543747a8787c1c7 already in progress: creating tablet
I20250625 13:52:08.060415 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet f033bfa4925b4d1c971789eb77efd92c (table default.Table_df8e6bfa3a4741fca89b09188c645be9 [id=e8ccbd9e2b83408b87689d6d6f26eb47]) successfully deleted
I20250625 13:52:08.062076 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 490c8902efeb46c99543747a8787c1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:08.062871 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 490c8902efeb46c99543747a8787c1c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 490c8902efeb46c99543747a8787c1c7 already in progress: creating tablet
I20250625 13:52:08.065165 1168 tablet_replica.cc:331] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.065361 1168 raft_consensus.cc:2241] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.065510 1168 raft_consensus.cc:2270] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.065968 1168 ts_tablet_manager.cc:1905] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.068038 1171 tablet_bootstrap.cc:654] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.068890 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 3b8e232f80724e38992aa5d66fd0caf3 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:08.070582 743 tablet_service.cc:1468] Processing CreateTablet for tablet 7d7b671a935b43a6a80a37d38e57f230 (DEFAULT_TABLE table=default.Table_5a6a094978314b53847a823ba43ac189 [id=dae6ef5d6bc5468da5b32bf934c386f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.070690 891 tablet_service.cc:1468] Processing CreateTablet for tablet 7d7b671a935b43a6a80a37d38e57f230 (DEFAULT_TABLE table=default.Table_5a6a094978314b53847a823ba43ac189 [id=dae6ef5d6bc5468da5b32bf934c386f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.071002 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7b671a935b43a6a80a37d38e57f230. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.071038 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7b671a935b43a6a80a37d38e57f230. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.075515 1170 tablet_bootstrap.cc:492] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.075698 1170 ts_tablet_manager.cc:1397] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.026s
I20250625 13:52:08.076210 1171 tablet_bootstrap.cc:492] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.076335 1171 ts_tablet_manager.cc:1397] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20250625 13:52:08.076433 1170 raft_consensus.cc:357] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.076640 1170 raft_consensus.cc:383] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.076732 1170 raft_consensus.cc:738] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.076975 1170 consensus_queue.cc:260] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.077019 1171 raft_consensus.cc:357] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.077183 1171 raft_consensus.cc:383] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.077225 1171 raft_consensus.cc:738] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.078119 1170 ts_tablet_manager.cc:1428] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:08.079115 1170 tablet_bootstrap.cc:492] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.079428 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 22bcbfcfeb0a439288622994b2d245b2 from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:08.079838 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 77772.1us
I20250625 13:52:08.080966 1171 consensus_queue.cc:260] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.081354 1170 tablet_bootstrap.cc:654] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.082335 1168 ts_tablet_manager.cc:1918] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.082475 1168 log.cc:1199] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/0e77736191544e8da32ca9927f0a5804
I20250625 13:52:08.082777 1168 ts_tablet_manager.cc:1939] T 0e77736191544e8da32ca9927f0a5804 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.083632 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 594) took 4613 ms. Trace:
I20250625 13:52:08.083720 1168 rpcz_store.cc:276] 0625 13:52:03.469855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.470292 (+ 437us) service_pool.cc:225] Handling call
0625 13:52:08.065115 (+4594823us) ts_tablet_manager.cc:1145] Deleting tablet 0e77736191544e8da32ca9927f0a5804
0625 13:52:08.065127 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.070750 (+ 5623us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.082242 (+ 11492us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.083612 (+ 1370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4586516}
I20250625 13:52:08.084123 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 588) took 4624 ms. Trace:
I20250625 13:52:08.084241 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 0e77736191544e8da32ca9927f0a5804 (table default.Table_b9f454bddd8a463b863eb0ca65ebf3af [id=7467e5b0513d42ff8c886b6080a5b539]) successfully deleted
I20250625 13:52:08.084201 1168 rpcz_store.cc:276] 0625 13:52:03.459707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.459930 (+ 223us) service_pool.cc:225] Handling call
0625 13:52:08.083998 (+4624068us) ts_tablet_manager.cc:1145] Deleting tablet 3016763b9e294b9da2861fe97e57e241
0625 13:52:08.084009 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.084107 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4609528}
I20250625 13:52:08.084524 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 589) took 4624 ms. Trace:
I20250625 13:52:08.084623 1168 rpcz_store.cc:276] 0625 13:52:03.460204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.460765 (+ 561us) service_pool.cc:225] Handling call
0625 13:52:08.084421 (+4623656us) ts_tablet_manager.cc:1145] Deleting tablet 93fdde7d068b400ba7070166eff63502
0625 13:52:08.084431 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.084509 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4607923}
I20250625 13:52:08.084887 1168 tablet_replica.cc:331] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:08.084884 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3016763b9e294b9da2861fe97e57e241 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3016763b9e294b9da2861fe97e57e241 already in progress: creating tablet
I20250625 13:52:08.085067 1168 raft_consensus.cc:2241] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.085206 1168 raft_consensus.cc:2270] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:08.085379 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 93fdde7d068b400ba7070166eff63502 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93fdde7d068b400ba7070166eff63502 already in progress: creating tablet
I20250625 13:52:08.086779 1171 ts_tablet_manager.cc:1428] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250625 13:52:08.086975 1168 ts_tablet_manager.cc:1905] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.091284 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 6c5b53c5d2514601b7bcf472ce2d1d84 (DEFAULT_TABLE table=default.Table_1402caa3f71f4bf7b40629b140616eeb [id=554fedd665a74496b66fd75021ed2611]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.091938 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c5b53c5d2514601b7bcf472ce2d1d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.092168 1171 tablet_bootstrap.cc:492] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.096828 1168 ts_tablet_manager.cc:1918] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.096961 1168 log.cc:1199] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/455884b62ee44b6da53543093f721027
I20250625 13:52:08.097261 1168 ts_tablet_manager.cc:1939] T 455884b62ee44b6da53543093f721027 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.097373 1171 tablet_bootstrap.cc:654] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.098937 743 tablet_service.cc:1468] Processing CreateTablet for tablet 6c5b53c5d2514601b7bcf472ce2d1d84 (DEFAULT_TABLE table=default.Table_1402caa3f71f4bf7b40629b140616eeb [id=554fedd665a74496b66fd75021ed2611]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.099447 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c5b53c5d2514601b7bcf472ce2d1d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.102109 1171 tablet_bootstrap.cc:492] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.102236 1171 ts_tablet_manager.cc:1397] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20250625 13:52:08.103204 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 608) took 4311 ms. Trace:
I20250625 13:52:08.103292 1168 rpcz_store.cc:276] 0625 13:52:03.791497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.791531 (+ 34us) service_pool.cc:225] Handling call
0625 13:52:08.084833 (+4293302us) ts_tablet_manager.cc:1145] Deleting tablet 455884b62ee44b6da53543093f721027
0625 13:52:08.084841 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.093627 (+ 8786us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.096732 (+ 3105us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.103182 (+ 6450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4293116}
I20250625 13:52:08.103731 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 609) took 4312 ms. Trace:
I20250625 13:52:08.103852 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 455884b62ee44b6da53543093f721027 (table default.Table_f51c5c5fe5394b598404b1944ee7d0b5 [id=67c521d398af4fc4bfe9e6c23d400906]) successfully deleted
I20250625 13:52:08.103814 1168 rpcz_store.cc:276] 0625 13:52:03.791632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.791754 (+ 122us) service_pool.cc:225] Handling call
0625 13:52:08.103612 (+4311858us) ts_tablet_manager.cc:1145] Deleting tablet 4d25e4114a184bc99705042593ab4ab9
0625 13:52:08.103623 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.103716 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4311651}
I20250625 13:52:08.104122 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 606) took 4313 ms. Trace:
I20250625 13:52:08.104192 1168 rpcz_store.cc:276] 0625 13:52:03.791179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.791246 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:08.104026 (+4312780us) ts_tablet_manager.cc:1145] Deleting tablet 3c6dd5b87ff34151836a3c29740e7c11
0625 13:52:08.104035 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.104108 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4308098}
W20250625 13:52:08.104414 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4d25e4114a184bc99705042593ab4ab9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d25e4114a184bc99705042593ab4ab9 already in progress: creating tablet
I20250625 13:52:08.104478 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 607) took 4313 ms. Trace:
I20250625 13:52:08.104533 1168 rpcz_store.cc:276] 0625 13:52:03.791389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.791431 (+ 42us) service_pool.cc:225] Handling call
0625 13:52:08.104386 (+4312955us) ts_tablet_manager.cc:1145] Deleting tablet 7db450dc63bd4b6caf074c67490cd175
0625 13:52:08.104396 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.104464 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4307634}
I20250625 13:52:08.104808 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 610) took 4313 ms. Trace:
I20250625 13:52:08.104871 1168 rpcz_store.cc:276] 0625 13:52:03.791761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.791999 (+ 238us) service_pool.cc:225] Handling call
0625 13:52:08.104720 (+4312721us) ts_tablet_manager.cc:1145] Deleting tablet be3313ec5dfb41b68662d3ff4c8c94f9
0625 13:52:08.104729 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.104795 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4307792}
I20250625 13:52:08.105110 1168 tablet_replica.cc:331] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:08.105075 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3c6dd5b87ff34151836a3c29740e7c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c6dd5b87ff34151836a3c29740e7c11 already in progress: creating tablet
I20250625 13:52:08.105276 1168 raft_consensus.cc:2241] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:08.105327 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 7db450dc63bd4b6caf074c67490cd175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db450dc63bd4b6caf074c67490cd175 already in progress: creating tablet
I20250625 13:52:08.105437 1168 raft_consensus.cc:2270] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:08.105784 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet be3313ec5dfb41b68662d3ff4c8c94f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be3313ec5dfb41b68662d3ff4c8c94f9 already in progress: creating tablet
I20250625 13:52:08.105927 1168 ts_tablet_manager.cc:1905] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.102909 1171 raft_consensus.cc:357] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.108122 1171 raft_consensus.cc:383] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.108201 1171 raft_consensus.cc:738] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.108713 1171 consensus_queue.cc:260] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.113430 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e1efe0a458494dc886f4dad68df5561f" } new_table_name: "default.Table_247151d8380c493883228e0cb0586a4b" modify_external_catalogs: true
I20250625 13:52:08.113806 1837 raft_consensus.cc:491] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.113658 891 tablet_service.cc:1468] Processing CreateTablet for tablet 6c5b53c5d2514601b7bcf472ce2d1d84 (DEFAULT_TABLE table=default.Table_1402caa3f71f4bf7b40629b140616eeb [id=554fedd665a74496b66fd75021ed2611]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.113921 1837 raft_consensus.cc:513] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.114228 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c5b53c5d2514601b7bcf472ce2d1d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.114451 1837 leader_election.cc:290] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:08.115031 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:52:08.115451 851 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.115906 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 455884b62ee44b6da53543093f721027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.118862 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.118947 1837 raft_consensus.cc:491] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.119061 1837 raft_consensus.cc:513] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:08.119426 849 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.119535 849 leader_election.cc:304] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.119602 1837 leader_election.cc:290] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:08.119977 1842 raft_consensus.cc:2747] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.120067 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d95a376304e464ab75e2e1e00e7a924" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:08.120231 774 raft_consensus.cc:2466] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.120355 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d95a376304e464ab75e2e1e00e7a924" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.120585 1046 raft_consensus.cc:2466] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.120689 851 leader_election.cc:304] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.120950 1842 raft_consensus.cc:2802] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.121016 1842 raft_consensus.cc:491] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.121076 1842 raft_consensus.cc:3058] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.122860 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b0037d09796455cbaaadfaa9c010673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.123085 892 tablet_service.cc:1515] Processing DeleteTablet for tablet d43ba17ebfec4244894d1b6e23200c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.123201 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdca99b734448118e0c42227c78e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.123667 1842 raft_consensus.cc:513] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.124429 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 3e60792b6796458c857415e258ece1c9 (DEFAULT_TABLE table=default.Table_0d7502063c8f4f06bcb3f4bd4c9e8cd9 [id=3a138811a1f94ae9ae8f2f7fa74bea64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.124886 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e60792b6796458c857415e258ece1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.124852 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d95a376304e464ab75e2e1e00e7a924" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:08.125036 1046 raft_consensus.cc:3058] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.125286 1168 ts_tablet_manager.cc:1918] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.125381 1168 log.cc:1199] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/6771d8e180fe48569a0e016db0cc9b51
I20250625 13:52:08.125664 1168 ts_tablet_manager.cc:1939] T 6771d8e180fe48569a0e016db0cc9b51 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.126327 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 611) took 4306 ms. Trace:
I20250625 13:52:08.126398 1168 rpcz_store.cc:276] 0625 13:52:03.820091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.820141 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:08.105068 (+4284927us) ts_tablet_manager.cc:1145] Deleting tablet 6771d8e180fe48569a0e016db0cc9b51
0625 13:52:08.105077 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.115488 (+ 10411us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.125183 (+ 9695us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.126308 (+ 1125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4284737}
I20250625 13:52:08.126695 1168 tablet_replica.cc:331] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.126857 1168 raft_consensus.cc:2241] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.126986 1168 raft_consensus.cc:2270] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.127406 1842 leader_election.cc:290] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:08.127461 1168 ts_tablet_manager.cc:1905] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.127889 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 6771d8e180fe48569a0e016db0cc9b51 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]) successfully deleted
I20250625 13:52:08.128297 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d95a376304e464ab75e2e1e00e7a924" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:08.128449 774 raft_consensus.cc:3058] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.131129 774 raft_consensus.cc:2466] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.131538 2018 raft_consensus.cc:491] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.131649 2018 raft_consensus.cc:513] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.132202 2018 leader_election.cc:290] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:08.132844 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad4ea8cf44944b08f53428c95809daf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.132956 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad4ea8cf44944b08f53428c95809daf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:08.133508 711 leader_election.cc:343] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:08.133709 711 leader_election.cc:343] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.133776 711 leader_election.cc:304] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:08.134217 2018 raft_consensus.cc:2747] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.134681 851 leader_election.cc:304] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.135186 1837 raft_consensus.cc:2802] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.135313 1837 raft_consensus.cc:695] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:08.135560 1837 consensus_queue.cc:237] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.136269 1046 raft_consensus.cc:2466] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.137925 1166 tablet_bootstrap.cc:492] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.138047 1166 ts_tablet_manager.cc:1397] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.165s user 0.007s sys 0.012s
I20250625 13:52:08.137913 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_07d09c7961df461ea4c3ee75152fd7a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.138576 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07d09c7961df461ea4c3ee75152fd7a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.138840 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3e60792b6796458c857415e258ece1c9 (DEFAULT_TABLE table=default.Table_0d7502063c8f4f06bcb3f4bd4c9e8cd9 [id=3a138811a1f94ae9ae8f2f7fa74bea64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.139199 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e60792b6796458c857415e258ece1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.139137 1166 raft_consensus.cc:357] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.139428 1166 raft_consensus.cc:383] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.139537 1166 raft_consensus.cc:738] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.139804 1166 consensus_queue.cc:260] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.142514 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8cf4529887f24853932e3c9a52a46b61" } new_table_name: "default.Table_3a4e5f10c2ec453a82fb97800a0d37f7" modify_external_catalogs: true
I20250625 13:52:08.142722 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_112725cbadf0437681278a5ff391dffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.143280 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_112725cbadf0437681278a5ff391dffe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.146162 1166 ts_tablet_manager.cc:1428] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250625 13:52:08.148285 1166 tablet_bootstrap.cc:492] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.148473 888 tablet_service.cc:1468] Processing CreateTablet for tablet 3e60792b6796458c857415e258ece1c9 (DEFAULT_TABLE table=default.Table_0d7502063c8f4f06bcb3f4bd4c9e8cd9 [id=3a138811a1f94ae9ae8f2f7fa74bea64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.148972 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e60792b6796458c857415e258ece1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.153975 1168 ts_tablet_manager.cc:1918] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.154086 1168 log.cc:1199] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/01ba67b500b646d7aa55315f7ee23ed6
I20250625 13:52:08.154346 1168 ts_tablet_manager.cc:1939] T 01ba67b500b646d7aa55315f7ee23ed6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.154625 1166 tablet_bootstrap.cc:654] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.155022 1171 ts_tablet_manager.cc:1428] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.053s user 0.001s sys 0.001s
I20250625 13:52:08.155045 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 612) took 4316 ms. Trace:
I20250625 13:52:08.158874 1168 rpcz_store.cc:276] 0625 13:52:03.838703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.838754 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:08.126654 (+4287900us) ts_tablet_manager.cc:1145] Deleting tablet 01ba67b500b646d7aa55315f7ee23ed6
0625 13:52:08.126663 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.130400 (+ 3737us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.153892 (+ 23492us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.155026 (+ 1134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4277433}
I20250625 13:52:08.159219 1168 tablet_replica.cc:331] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.159368 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 01ba67b500b646d7aa55315f7ee23ed6 (table default.Table_e1efe0a458494dc886f4dad68df5561f [id=1ccdf11a8fe64810a774e47fb27d0916]) successfully deleted
I20250625 13:52:08.159535 1171 tablet_bootstrap.cc:492] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.159945 1168 raft_consensus.cc:2241] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.160431 1168 raft_consensus.cc:2270] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.161015 1168 ts_tablet_manager.cc:1905] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.161240 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a6f4712c89364336820178423f4301a8 (DEFAULT_TABLE table=default.Table_80f1e7ae1dce4246b10319e14d4dcaa6 [id=baa42ee80099427882f9ff2fea8ad26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.161803 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f4712c89364336820178423f4301a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.165829 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.166083 1165 tablet_replica.cc:331] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:08.166252 1165 raft_consensus.cc:2241] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.166393 1165 raft_consensus.cc:2270] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.167013 1165 ts_tablet_manager.cc:1905] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.167637 595 catalog_manager.cc:5582] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:08.168066 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:08.169064 2050 tablet.cc:1722] T d8b30c229c3445b683a5015f069eca1c P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:08.174337 1165 ts_tablet_manager.cc:1918] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.174755 1165 log.cc:1199] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/87f5b36ddb474ed6903c14876b236ba8
I20250625 13:52:08.175122 1165 ts_tablet_manager.cc:1939] T 87f5b36ddb474ed6903c14876b236ba8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:08.177256 1171 tablet_bootstrap.cc:654] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.178311 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 87f5b36ddb474ed6903c14876b236ba8 (table default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]) successfully deleted
I20250625 13:52:08.179652 1168 ts_tablet_manager.cc:1918] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.179769 1168 log.cc:1199] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/9a31f10c8ca745b7ad635c622a265450
I20250625 13:52:08.180066 1168 ts_tablet_manager.cc:1939] T 9a31f10c8ca745b7ad635c622a265450 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.181828 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 623) took 4268 ms. Trace:
I20250625 13:52:08.181921 1168 rpcz_store.cc:276] 0625 13:52:03.913769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.916805 (+ 3036us) service_pool.cc:225] Handling call
0625 13:52:08.159174 (+4242369us) ts_tablet_manager.cc:1145] Deleting tablet 9a31f10c8ca745b7ad635c622a265450
0625 13:52:08.159183 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.168240 (+ 9057us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.179537 (+ 11297us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.181808 (+ 2271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4242080}
I20250625 13:52:08.182255 1168 tablet_replica.cc:331] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.182401 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 9a31f10c8ca745b7ad635c622a265450 (table default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]) successfully deleted
I20250625 13:52:08.182451 1168 raft_consensus.cc:2241] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.182591 1168 raft_consensus.cc:2270] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.183087 1168 ts_tablet_manager.cc:1905] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.184114 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet d8b30c229c3445b683a5015f069eca1c (table default.Table_d6bab22e154a444f90a895591fa09ca3 [id=5ad3bfe6145f4a4cbc7dc4a31cc43507]). Expected version 3 got 0
I20250625 13:52:08.186481 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:08.187065 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 87f5b36ddb474ed6903c14876b236ba8 because the tablet was not found. No further retry: Not found: Tablet not found: 87f5b36ddb474ed6903c14876b236ba8
I20250625 13:52:08.187521 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1dc56df21cb14dd1b0c006c14bf997e6" } new_table_name: "default.Table_eef69b729d1a490a8704b781d33e575e" modify_external_catalogs: true
I20250625 13:52:08.188093 1166 tablet_bootstrap.cc:492] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.188246 1166 ts_tablet_manager.cc:1397] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.003s
I20250625 13:52:08.188995 1166 raft_consensus.cc:357] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.189162 1166 raft_consensus.cc:383] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.189230 1166 raft_consensus.cc:738] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.189464 1166 consensus_queue.cc:260] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.195253 751 tablet_service.cc:1468] Processing CreateTablet for tablet a6f4712c89364336820178423f4301a8 (DEFAULT_TABLE table=default.Table_80f1e7ae1dce4246b10319e14d4dcaa6 [id=baa42ee80099427882f9ff2fea8ad26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.195602 1026 tablet_service.cc:1468] Processing CreateTablet for tablet eafa481b5ea04aa4a6a478239fa8c91b (DEFAULT_TABLE table=default.Table_fa6f50b15eff4238819930c9bf68b192 [id=f12933ffcf334a3f9385df523bcc02e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.195745 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f4712c89364336820178423f4301a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.195986 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafa481b5ea04aa4a6a478239fa8c91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.196254 752 tablet_service.cc:1468] Processing CreateTablet for tablet eafa481b5ea04aa4a6a478239fa8c91b (DEFAULT_TABLE table=default.Table_fa6f50b15eff4238819930c9bf68b192 [id=f12933ffcf334a3f9385df523bcc02e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.196609 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafa481b5ea04aa4a6a478239fa8c91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.202462 1166 ts_tablet_manager.cc:1428] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
W20250625 13:52:08.204577 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5565f43d293e4e809d6aff72a201db3e consensus state has no leader
I20250625 13:52:08.205191 888 tablet_service.cc:1468] Processing CreateTablet for tablet eafa481b5ea04aa4a6a478239fa8c91b (DEFAULT_TABLE table=default.Table_fa6f50b15eff4238819930c9bf68b192 [id=f12933ffcf334a3f9385df523bcc02e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.205559 1168 ts_tablet_manager.cc:1918] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.205650 1166 tablet_bootstrap.cc:492] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.205691 1168 log.cc:1199] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/f00e72e103f443e886f0df690cff4b6f
I20250625 13:52:08.205883 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eafa481b5ea04aa4a6a478239fa8c91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.208943 1168 ts_tablet_manager.cc:1939] T f00e72e103f443e886f0df690cff4b6f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.209676 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 624) took 4266 ms. Trace:
I20250625 13:52:08.209762 1168 rpcz_store.cc:276] 0625 13:52:03.943463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:03.943520 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:08.182209 (+4238689us) ts_tablet_manager.cc:1145] Deleting tablet f00e72e103f443e886f0df690cff4b6f
0625 13:52:08.182219 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.186111 (+ 3892us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.205458 (+ 19347us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.209652 (+ 4194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4238438}
I20250625 13:52:08.210163 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 635) took 3848 ms. Trace:
I20250625 13:52:08.210251 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet f00e72e103f443e886f0df690cff4b6f (table default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]) successfully deleted
I20250625 13:52:08.210237 1168 rpcz_store.cc:276] 0625 13:52:04.361437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.361851 (+ 414us) service_pool.cc:225] Handling call
0625 13:52:08.210041 (+3848190us) ts_tablet_manager.cc:1145] Deleting tablet eabd67b9ceb74bd1a5f4f115c7db0e87
0625 13:52:08.210052 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.210146 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3813097}
I20250625 13:52:08.210592 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 636) took 3763 ms. Trace:
I20250625 13:52:08.210665 1168 rpcz_store.cc:276] 0625 13:52:04.447750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.447865 (+ 115us) service_pool.cc:225] Handling call
0625 13:52:08.210480 (+3762615us) ts_tablet_manager.cc:1145] Deleting tablet e3b3d5a2ab7042c2a2f18890363e7a9b
0625 13:52:08.210488 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.210565 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3755138}
W20250625 13:52:08.210775 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet eabd67b9ceb74bd1a5f4f115c7db0e87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eabd67b9ceb74bd1a5f4f115c7db0e87 already in progress: creating tablet
I20250625 13:52:08.210958 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 638) took 3762 ms. Trace:
I20250625 13:52:08.211021 1168 rpcz_store.cc:276] 0625 13:52:04.448935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.449016 (+ 81us) service_pool.cc:225] Handling call
0625 13:52:08.210864 (+3761848us) ts_tablet_manager.cc:1145] Deleting tablet c737daaede054d09a0c07564157423a9
0625 13:52:08.210871 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.210940 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3755143}
I20250625 13:52:08.211352 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 639) took 3761 ms. Trace:
W20250625 13:52:08.211347 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e3b3d5a2ab7042c2a2f18890363e7a9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b3d5a2ab7042c2a2f18890363e7a9b already in progress: creating tablet
W20250625 13:52:08.211833 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c737daaede054d09a0c07564157423a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c737daaede054d09a0c07564157423a9 already in progress: creating tablet
I20250625 13:52:08.211419 1168 rpcz_store.cc:276] 0625 13:52:04.449427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.449510 (+ 83us) service_pool.cc:225] Handling call
0625 13:52:08.211217 (+3761707us) ts_tablet_manager.cc:1145] Deleting tablet eafc402d809245d189f3977162506ef8
0625 13:52:08.211227 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.211297 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3755467}
I20250625 13:52:08.214974 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 637) took 3766 ms. Trace:
I20250625 13:52:08.215049 1168 rpcz_store.cc:276] 0625 13:52:04.448436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.448534 (+ 98us) service_pool.cc:225] Handling call
0625 13:52:08.214874 (+3766340us) ts_tablet_manager.cc:1145] Deleting tablet d7660a2b70744bd589220b28e6edbc42
0625 13:52:08.214883 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.214959 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3695154}
W20250625 13:52:08.215118 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet eafc402d809245d189f3977162506ef8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eafc402d809245d189f3977162506ef8 already in progress: creating tablet
I20250625 13:52:08.215212 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3b8e232f80724e38992aa5d66fd0caf3
I20250625 13:52:08.216001 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 645) took 3720 ms. Trace:
I20250625 13:52:08.216068 1168 rpcz_store.cc:276] 0625 13:52:04.495631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.498558 (+ 2927us) service_pool.cc:225] Handling call
0625 13:52:08.215253 (+3716695us) ts_tablet_manager.cc:1145] Deleting tablet 067272869c5547648c90f83a34a036aa
0625 13:52:08.215262 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.215985 (+ 723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3691765}
I20250625 13:52:08.216363 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 646) took 3720 ms. Trace:
I20250625 13:52:08.216426 1168 rpcz_store.cc:276] 0625 13:52:04.495745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.519596 (+ 23851us) service_pool.cc:225] Handling call
0625 13:52:08.216274 (+3696678us) ts_tablet_manager.cc:1145] Deleting tablet 8ebe47e363ce4a6c8129169bc6777e63
0625 13:52:08.216282 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.216351 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3685157}
I20250625 13:52:08.216666 1168 tablet_replica.cc:331] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.216692 891 tablet_service.cc:1468] Processing CreateTablet for tablet a6f4712c89364336820178423f4301a8 (DEFAULT_TABLE table=default.Table_80f1e7ae1dce4246b10319e14d4dcaa6 [id=baa42ee80099427882f9ff2fea8ad26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.216848 1168 raft_consensus.cc:2241] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.216987 1168 raft_consensus.cc:2270] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.217190 594 catalog_manager.cc:6652] ReplaceTablet: tablet 3b8e232f80724e38992aa5d66fd0caf3 deleted and replaced by tablet 4181bbb4af9c4141b587968cb7ef8251
I20250625 13:52:08.217211 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f4712c89364336820178423f4301a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.217486 1168 ts_tablet_manager.cc:1905] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:08.215696 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d7660a2b70744bd589220b28e6edbc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7660a2b70744bd589220b28e6edbc42 already in progress: creating tablet
W20250625 13:52:08.220324 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 067272869c5547648c90f83a34a036aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 067272869c5547648c90f83a34a036aa already in progress: creating tablet
I20250625 13:52:08.220444 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b8e232f80724e38992aa5d66fd0caf3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4181bbb4af9c4141b587968cb7ef8251) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:08.220625 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8ebe47e363ce4a6c8129169bc6777e63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ebe47e363ce4a6c8129169bc6777e63 already in progress: creating tablet
I20250625 13:52:08.223198 1168 ts_tablet_manager.cc:1918] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.221801 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0568e1a3b89f4380abb0d3897817ccef" } new_table_name: "default.Table_4a226ca5c5c84ef380d6c37dd08f6740" modify_external_catalogs: true
I20250625 13:52:08.225181 1168 log.cc:1199] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/46316eb725bc47e695af476c0b185504
I20250625 13:52:08.225699 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b8e232f80724e38992aa5d66fd0caf3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4181bbb4af9c4141b587968cb7ef8251) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.226090 1170 tablet_bootstrap.cc:492] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.226240 1170 ts_tablet_manager.cc:1397] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.147s user 0.018s sys 0.034s
W20250625 13:52:08.226723 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3b8e232f80724e38992aa5d66fd0caf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b8e232f80724e38992aa5d66fd0caf3 already in progress: creating tablet
I20250625 13:52:08.227025 1170 raft_consensus.cc:357] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.227210 1170 raft_consensus.cc:383] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.227279 1170 raft_consensus.cc:738] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.227190 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ae346c0db6734c4684eb50013dda4b4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.227545 1170 consensus_queue.cc:260] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:08.227917 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae346c0db6734c4684eb50013dda4b4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.228019 1170 ts_tablet_manager.cc:1428] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:08.228317 1170 tablet_bootstrap.cc:492] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.229848 1170 tablet_bootstrap.cc:654] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.232707 892 tablet_service.cc:1468] Processing CreateTablet for tablet d18f80d9a51f4afbadfa239fe28a92ca (DEFAULT_TABLE table=default.Table_08d0c030c2cd4f46ad44c12bfee6842d [id=29646aaff9444c0ca835b4722d08875e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.233083 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18f80d9a51f4afbadfa239fe28a92ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.235702 1168 ts_tablet_manager.cc:1939] T 46316eb725bc47e695af476c0b185504 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.236011 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 22bcbfcfeb0a439288622994b2d245b2
I20250625 13:52:08.236425 592 catalog_manager.cc:6652] ReplaceTablet: tablet 22bcbfcfeb0a439288622994b2d245b2 deleted and replaced by tablet 34fe02d7b37f4d99bbae082943223e35
I20250625 13:52:08.237499 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 647) took 3741 ms. Trace:
I20250625 13:52:08.237921 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bcbfcfeb0a439288622994b2d245b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 34fe02d7b37f4d99bbae082943223e35) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.237957 1168 rpcz_store.cc:276] 0625 13:52:04.495996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.496317 (+ 321us) service_pool.cc:225] Handling call
0625 13:52:08.216626 (+3720309us) ts_tablet_manager.cc:1145] Deleting tablet 46316eb725bc47e695af476c0b185504
0625 13:52:08.216634 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.220335 (+ 3701us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.223116 (+ 2781us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.237480 (+ 14364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3682902}
I20250625 13:52:08.238062 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f3b3b6035e8e4cf9872da9d67e09e7ae" } modify_external_catalogs: true
I20250625 13:52:08.238268 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f3b3b6035e8e4cf9872da9d67e09e7ae" } modify_external_catalogs: true
I20250625 13:52:08.238308 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 648) took 3654 ms. Trace:
I20250625 13:52:08.238369 1168 rpcz_store.cc:276] 0625 13:52:04.583750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:04.585948 (+ 2198us) service_pool.cc:225] Handling call
0625 13:52:08.238202 (+3652254us) ts_tablet_manager.cc:1145] Deleting tablet e8d2ab8f91a045a3a39f3d09459f2a28
0625 13:52:08.238211 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.238296 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3652043}
I20250625 13:52:08.238615 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 655) took 2877 ms. Trace:
I20250625 13:52:08.238667 1168 rpcz_store.cc:276] 0625 13:52:05.361521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.361581 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:08.238532 (+2876951us) ts_tablet_manager.cc:1145] Deleting tablet 254afffd283449fdaf5e190ba3028f16
0625 13:52:08.238539 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.238605 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2876702}
I20250625 13:52:08.238926 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 657) took 2856 ms. Trace:
I20250625 13:52:08.238986 1168 rpcz_store.cc:276] 0625 13:52:05.382003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.382060 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:08.238833 (+2856773us) ts_tablet_manager.cc:1145] Deleting tablet b7d888e628a94f939ecf2571e60b1948
0625 13:52:08.238842 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.238913 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2856548}
I20250625 13:52:08.239269 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 656) took 2858 ms. Trace:
I20250625 13:52:08.239365 1168 rpcz_store.cc:276] 0625 13:52:05.380630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.384409 (+ 3779us) service_pool.cc:225] Handling call
0625 13:52:08.239170 (+2854761us) ts_tablet_manager.cc:1145] Deleting tablet 1a6d0dbbe2894e61b0ea08f98bce61e7
0625 13:52:08.239180 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.239254 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2854570}
I20250625 13:52:08.239579 1168 tablet_replica.cc:331] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.239732 1168 raft_consensus.cc:2241] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.240413 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bcbfcfeb0a439288622994b2d245b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 34fe02d7b37f4d99bbae082943223e35) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.241636 1171 tablet_bootstrap.cc:492] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.241752 1171 ts_tablet_manager.cc:1397] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.022s
I20250625 13:52:08.242287 1168 raft_consensus.cc:2270] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.242501 1171 raft_consensus.cc:357] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.242666 1171 raft_consensus.cc:383] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.242730 1171 raft_consensus.cc:738] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.243481 1170 tablet_bootstrap.cc:492] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.243645 1170 ts_tablet_manager.cc:1397] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20250625 13:52:08.243834 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bcbfcfeb0a439288622994b2d245b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 34fe02d7b37f4d99bbae082943223e35) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.244220 751 tablet_service.cc:1468] Processing CreateTablet for tablet d18f80d9a51f4afbadfa239fe28a92ca (DEFAULT_TABLE table=default.Table_08d0c030c2cd4f46ad44c12bfee6842d [id=29646aaff9444c0ca835b4722d08875e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:08.244330 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e8d2ab8f91a045a3a39f3d09459f2a28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8d2ab8f91a045a3a39f3d09459f2a28 already in progress: creating tablet
W20250625 13:52:08.244571 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 254afffd283449fdaf5e190ba3028f16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 254afffd283449fdaf5e190ba3028f16 already in progress: creating tablet
I20250625 13:52:08.244659 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18f80d9a51f4afbadfa239fe28a92ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:08.244791 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet b7d888e628a94f939ecf2571e60b1948 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7d888e628a94f939ecf2571e60b1948 already in progress: creating tablet
I20250625 13:52:08.244426 1170 raft_consensus.cc:357] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.245015 1170 raft_consensus.cc:383] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:52:08.245012 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a6d0dbbe2894e61b0ea08f98bce61e7 already in progress: creating tablet
I20250625 13:52:08.245093 1170 raft_consensus.cc:738] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:08.245218 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 22bcbfcfeb0a439288622994b2d245b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bcbfcfeb0a439288622994b2d245b2 already in progress: creating tablet
I20250625 13:52:08.245355 1170 consensus_queue.cc:260] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:08.247077 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 consensus state has no leader
I20250625 13:52:08.247113 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b8e232f80724e38992aa5d66fd0caf3 with delete_type TABLET_DATA_DELETED (replaced by tablet 4181bbb4af9c4141b587968cb7ef8251) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.247855 1170 ts_tablet_manager.cc:1428] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:08.249094 1170 tablet_bootstrap.cc:492] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.249958 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_906b9e1a3a8147fb935f7a96968ed294" } modify_external_catalogs: true
I20250625 13:52:08.250145 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_906b9e1a3a8147fb935f7a96968ed294" } modify_external_catalogs: true
I20250625 13:52:08.242830 1168 ts_tablet_manager.cc:1905] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.250751 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d18f80d9a51f4afbadfa239fe28a92ca (DEFAULT_TABLE table=default.Table_08d0c030c2cd4f46ad44c12bfee6842d [id=29646aaff9444c0ca835b4722d08875e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.251235 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18f80d9a51f4afbadfa239fe28a92ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.251449 1166 tablet_bootstrap.cc:654] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:08.252880 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 22bcbfcfeb0a439288622994b2d245b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bcbfcfeb0a439288622994b2d245b2 already in progress: creating tablet
W20250625 13:52:08.253142 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3b8e232f80724e38992aa5d66fd0caf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b8e232f80724e38992aa5d66fd0caf3 already in progress: creating tablet
I20250625 13:52:08.253337 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 9f3212d0b8de440e82156323ecb40a7d (DEFAULT_TABLE table=default.Table_3e4c0ca74c2f4252a4dc3c7e85afc3d0 [id=66593cc257364798926b6c146e244409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.253796 1170 tablet_bootstrap.cc:654] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.254163 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3212d0b8de440e82156323ecb40a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.242944 1171 consensus_queue.cc:260] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.258005 1171 ts_tablet_manager.cc:1428] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250625 13:52:08.264142 1168 ts_tablet_manager.cc:1918] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.264263 1168 log.cc:1199] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/991bb428def545fd979e9f1a80ce4d18
I20250625 13:52:08.264542 1168 ts_tablet_manager.cc:1939] T 991bb428def545fd979e9f1a80ce4d18 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.265174 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c37f5cca4df4a929018f7a81e5cdcb3" } modify_external_catalogs: true
I20250625 13:52:08.265349 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c37f5cca4df4a929018f7a81e5cdcb3" } modify_external_catalogs: true
I20250625 13:52:08.265828 1837 raft_consensus.cc:491] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.266000 1837 raft_consensus.cc:513] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.266741 1837 leader_election.cc:290] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:08.268539 851 leader_election.cc:343] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1effed0fa10d409abcb31dbba54861d6
I20250625 13:52:08.269117 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1effed0fa10d409abcb31dbba54861d6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.269454 1045 raft_consensus.cc:2466] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.269945 849 leader_election.cc:304] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.270414 1837 raft_consensus.cc:2802] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.270498 1837 raft_consensus.cc:491] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.270574 1837 raft_consensus.cc:3058] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.272547 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4187a17a1e4245cf8d7d644bb7713836" } modify_external_catalogs: true
I20250625 13:52:08.272738 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4187a17a1e4245cf8d7d644bb7713836" } modify_external_catalogs: true
W20250625 13:52:08.273531 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef2b08fba37f4a38880c018ec2d869c8 consensus state has no leader
I20250625 13:52:08.273905 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b23f837492294813bff5281ce11e1b9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.274116 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e99b7145ea274ef2a8d4bef4fb4e16ad has no consensus state
W20250625 13:52:08.274492 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b23f837492294813bff5281ce11e1b9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.275061 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b88679cb7cc74587927204b82e5915f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.275496 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8cc353b88a124fac801b8898b5a5f72c
I20250625 13:52:08.277101 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet dc75a0a8421b457089cc707d9eb6b76f (table default.Table_f618b45e019044ea946538ed3fa05471 [id=fa0246aff3f648538e6fddcb30f3a399]). Expected version 3 got 0
I20250625 13:52:08.277314 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:08.277418 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c3e15d16f1e74232b39b0c990c64bb6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.281801 1837 raft_consensus.cc:513] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:08.282508 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3e15d16f1e74232b39b0c990c64bb6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:08.283012 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a2007e51d424053bc2836dedf46add2 consensus state has no leader
I20250625 13:52:08.283828 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1effed0fa10d409abcb31dbba54861d6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:08.284050 1048 raft_consensus.cc:3058] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.284008 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:08.284582 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_47598e6b1611413784a8c9ab9d1c0a67" } modify_external_catalogs: true
I20250625 13:52:08.284757 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_47598e6b1611413784a8c9ab9d1c0a67" } modify_external_catalogs: true
W20250625 13:52:08.286036 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b88679cb7cc74587927204b82e5915f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.286993 1048 raft_consensus.cc:2466] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.288710 849 leader_election.cc:304] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.289080 1842 raft_consensus.cc:2802] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.289239 1842 raft_consensus.cc:695] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:08.289507 1842 consensus_queue.cc:237] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.292562 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cc353b88a124fac801b8898b5a5f72c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.292843 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 36b151e200e1430c99ac194bc55881c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.293017 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet abd6f9121a9d4b39adde1b880a875e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.293399 1710 tablet_replica.cc:331] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:08.293614 1710 raft_consensus.cc:2241] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.293782 1710 raft_consensus.cc:2270] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:08.294394 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8cc353b88a124fac801b8898b5a5f72c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cc353b88a124fac801b8898b5a5f72c already in progress: creating tablet
I20250625 13:52:08.294497 1710 ts_tablet_manager.cc:1905] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.294754 1837 leader_election.cc:290] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.294963 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cc353b88a124fac801b8898b5a5f72c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:08.295694 851 leader_election.cc:343] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1effed0fa10d409abcb31dbba54861d6
I20250625 13:52:08.299100 1171 tablet_bootstrap.cc:492] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.299437 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 455884b62ee44b6da53543093f721027 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.299650 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 1effed0fa10d409abcb31dbba54861d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.300698 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cc353b88a124fac801b8898b5a5f72c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:08.301441 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8cc353b88a124fac801b8898b5a5f72c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cc353b88a124fac801b8898b5a5f72c already in progress: creating tablet
I20250625 13:52:08.301522 1171 tablet_bootstrap.cc:654] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.301496 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 668) took 2806 ms. Trace:
I20250625 13:52:08.301594 1168 rpcz_store.cc:276] 0625 13:52:05.495236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.495287 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:08.239535 (+2744248us) ts_tablet_manager.cc:1145] Deleting tablet 991bb428def545fd979e9f1a80ce4d18
0625 13:52:08.239542 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.259398 (+ 19856us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.264051 (+ 4653us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.301475 (+ 37424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2744022}
I20250625 13:52:08.301973 1168 tablet_replica.cc:331] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.302119 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 991bb428def545fd979e9f1a80ce4d18 (table default.Table_a21873ae3bbd4a1aaf287e80fe14c12e [id=fd9a1350f16e4c54a613cf8f7026240a]) successfully deleted
I20250625 13:52:08.302155 1168 raft_consensus.cc:2241] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.302286 1168 raft_consensus.cc:2270] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.302794 1168 ts_tablet_manager.cc:1905] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.304773 892 tablet_service.cc:1468] Processing CreateTablet for tablet 9f3212d0b8de440e82156323ecb40a7d (DEFAULT_TABLE table=default.Table_3e4c0ca74c2f4252a4dc3c7e85afc3d0 [id=66593cc257364798926b6c146e244409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.305927 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9f3212d0b8de440e82156323ecb40a7d (DEFAULT_TABLE table=default.Table_3e4c0ca74c2f4252a4dc3c7e85afc3d0 [id=66593cc257364798926b6c146e244409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.306285 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3212d0b8de440e82156323ecb40a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.310938 1710 ts_tablet_manager.cc:1918] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.311296 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 063a793e0d7747668d2728fdc2160b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.313714 1168 ts_tablet_manager.cc:1918] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.313882 1168 log.cc:1199] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/54fa042947da4e1b89018640b46b271a
I20250625 13:52:08.314218 1168 ts_tablet_manager.cc:1939] T 54fa042947da4e1b89018640b46b271a P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.314368 1710 log.cc:1199] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/36b151e200e1430c99ac194bc55881c1
I20250625 13:52:08.314757 1710 ts_tablet_manager.cc:1939] T 36b151e200e1430c99ac194bc55881c1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:08.315156 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 066db01f791542a38cfb6e408c4836ed
I20250625 13:52:08.317251 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet da6bcb2055b94749879607559e62f6eb
I20250625 13:52:08.317649 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 066db01f791542a38cfb6e408c4836ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.318333 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3212d0b8de440e82156323ecb40a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.318403 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 066db01f791542a38cfb6e408c4836ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.318819 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 066db01f791542a38cfb6e408c4836ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:08.319151 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 066db01f791542a38cfb6e408c4836ed because the tablet was not found. No further retry: Not found: Tablet not found: 066db01f791542a38cfb6e408c4836ed
I20250625 13:52:08.319344 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet da6bcb2055b94749879607559e62f6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.319934 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d0f8fc1edba14555a44cee33f16218ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.323096 881 tablet_service.cc:1515] Processing DeleteTablet for tablet da6bcb2055b94749879607559e62f6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:08.323173 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0f8fc1edba14555a44cee33f16218ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.323482 752 tablet_service.cc:1515] Processing DeleteTablet for tablet da6bcb2055b94749879607559e62f6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.323589 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 36b151e200e1430c99ac194bc55881c1 (table default.Table_29924a11586f4d2bb3d0f8574a7849f8 [id=e057b2ec6f914a01b474481bc5c25349]) successfully deleted
W20250625 13:52:08.324194 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet da6bcb2055b94749879607559e62f6eb because the tablet was not found. No further retry: Not found: Tablet not found: da6bcb2055b94749879607559e62f6eb
I20250625 13:52:08.324445 1710 tablet_replica.cc:331] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:08.324672 1710 raft_consensus.cc:2241] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.325043 1710 raft_consensus.cc:2270] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.325439 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5cf73b4ab99c466abcfedf1b200e743b
I20250625 13:52:08.325764 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 670) took 2802 ms. Trace:
I20250625 13:52:08.325853 1168 rpcz_store.cc:276] 0625 13:52:05.523749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.523817 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:08.301930 (+2778113us) ts_tablet_manager.cc:1145] Deleting tablet 54fa042947da4e1b89018640b46b271a
0625 13:52:08.301939 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.305699 (+ 3760us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.313626 (+ 7927us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.325745 (+ 12119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2777838}
I20250625 13:52:08.326334 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c0f8e222d6aa4f5582688f874ce04f4f
I20250625 13:52:08.326333 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cf73b4ab99c466abcfedf1b200e743b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.326660 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cf73b4ab99c466abcfedf1b200e743b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.326900 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 673) took 2802 ms. Trace:
I20250625 13:52:08.327001 1168 rpcz_store.cc:276] 0625 13:52:05.524155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.524196 (+ 41us) service_pool.cc:225] Handling call
0625 13:52:08.326544 (+2802348us) ts_tablet_manager.cc:1145] Deleting tablet 00d4acd387c84d55834f632f41045862
0625 13:52:08.326554 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.326886 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2802158}
I20250625 13:52:08.327095 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f8e222d6aa4f5582688f874ce04f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.327370 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 54fa042947da4e1b89018640b46b271a (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]) successfully deleted
I20250625 13:52:08.327921 746 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f8e222d6aa4f5582688f874ce04f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.328532 1710 ts_tablet_manager.cc:1905] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.328855 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 675) took 2804 ms. Trace:
I20250625 13:52:08.328969 1168 rpcz_store.cc:276] 0625 13:52:05.524434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.524481 (+ 47us) service_pool.cc:225] Handling call
0625 13:52:08.328752 (+2804271us) ts_tablet_manager.cc:1145] Deleting tablet c4db530b54f04fedb18bf993dc60d8da
0625 13:52:08.328761 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.328839 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2804110}
I20250625 13:52:08.329346 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 672) took 2805 ms. Trace:
W20250625 13:52:08.333178 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c0f8e222d6aa4f5582688f874ce04f4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f8e222d6aa4f5582688f874ce04f4f already in progress: creating tablet
I20250625 13:52:08.333668 1168 rpcz_store.cc:276] 0625 13:52:05.524018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.524761 (+ 743us) service_pool.cc:225] Handling call
0625 13:52:08.329250 (+2804489us) ts_tablet_manager.cc:1145] Deleting tablet da2f8d0d9a714a28ba6fb9642d461de2
0625 13:52:08.329260 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.329333 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2804323}
I20250625 13:52:08.334084 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 674) took 2809 ms. Trace:
I20250625 13:52:08.334209 1168 rpcz_store.cc:276] 0625 13:52:05.524305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.524963 (+ 658us) service_pool.cc:225] Handling call
0625 13:52:08.333983 (+2809020us) ts_tablet_manager.cc:1145] Deleting tablet 58a532e889244b088562081e444234a3
0625 13:52:08.333992 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.334070 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2808872}
I20250625 13:52:08.333564 1842 raft_consensus.cc:491] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.334476 1842 raft_consensus.cc:513] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.335707 1842 leader_election.cc:290] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.336007 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "063a793e0d7747668d2728fdc2160b5e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.336331 881 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f8e222d6aa4f5582688f874ce04f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:08.336508 849 leader_election.cc:343] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.332556 1171 tablet_bootstrap.cc:492] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.336714 1166 tablet_bootstrap.cc:492] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.336797 1171 ts_tablet_manager.cc:1397] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250625 13:52:08.336962 1166 ts_tablet_manager.cc:1397] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.131s user 0.000s sys 0.013s
W20250625 13:52:08.337086 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 00d4acd387c84d55834f632f41045862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00d4acd387c84d55834f632f41045862 already in progress: creating tablet
W20250625 13:52:08.337476 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet da2f8d0d9a714a28ba6fb9642d461de2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da2f8d0d9a714a28ba6fb9642d461de2 already in progress: creating tablet
I20250625 13:52:08.337621 1171 raft_consensus.cc:357] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.337777 1171 raft_consensus.cc:383] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.337836 1171 raft_consensus.cc:738] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.337780 1166 raft_consensus.cc:357] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.338003 1166 raft_consensus.cc:383] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.338073 1166 raft_consensus.cc:738] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.338266 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cf73b4ab99c466abcfedf1b200e743b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.338348 1166 consensus_queue.cc:260] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.338563 1165 tablet_replica.cc:331] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:08.338752 1168 tablet_replica.cc:331] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.338732 1165 raft_consensus.cc:2241] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.338871 1165 raft_consensus.cc:2270] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.338913 1168 raft_consensus.cc:2241] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:08.339226 1168 raft_consensus.cc:2270] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.339401 1165 ts_tablet_manager.cc:1905] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.339774 1168 ts_tablet_manager.cc:1905] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.340196 1171 consensus_queue.cc:260] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.343102 1166 ts_tablet_manager.cc:1428] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250625 13:52:08.343217 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "063a793e0d7747668d2728fdc2160b5e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:08.343390 1166 tablet_bootstrap.cc:492] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:08.346292 851 leader_election.cc:343] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.346434 851 leader_election.cc:304] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.346314 1171 ts_tablet_manager.cc:1428] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 13:52:08.346810 1842 raft_consensus.cc:2747] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.347159 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.347723 1171 tablet_bootstrap.cc:492] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.349731 1165 ts_tablet_manager.cc:1918] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.349915 1165 log.cc:1199] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/5cf73b4ab99c466abcfedf1b200e743b
I20250625 13:52:08.350241 1171 tablet_bootstrap.cc:654] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.350351 1165 ts_tablet_manager.cc:1939] T 5cf73b4ab99c466abcfedf1b200e743b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:08.351699 1165 tablet_replica.cc:331] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:08.351675 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_efed4c6940fa46909feec8a3c6f600d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.351915 1165 raft_consensus.cc:2241] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.352041 1165 raft_consensus.cc:2270] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:08.352299 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 58a532e889244b088562081e444234a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58a532e889244b088562081e444234a3 already in progress: creating tablet
W20250625 13:52:08.352303 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efed4c6940fa46909feec8a3c6f600d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.354780 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 753e4767d7b84515ad2c93437e46c65b (DEFAULT_TABLE table=default.Table_3a79d0adb2204bcaa209ad1de0fa64fc [id=ad7755e4d3614d179c349c6c3ceb9d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.355201 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753e4767d7b84515ad2c93437e46c65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.356381 1710 ts_tablet_manager.cc:1918] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.356489 1710 log.cc:1199] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/abd6f9121a9d4b39adde1b880a875e23
I20250625 13:52:08.356787 1710 ts_tablet_manager.cc:1939] T abd6f9121a9d4b39adde1b880a875e23 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:08.357224 1710 tablet_replica.cc:331] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:08.357429 1710 raft_consensus.cc:2241] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.357594 1710 raft_consensus.cc:2270] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.351459 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 5cf73b4ab99c466abcfedf1b200e743b (table default.Table_47598e6b1611413784a8c9ab9d1c0a67 [id=8c0cf5af3aee43e9b7676c1be4842093]) successfully deleted
I20250625 13:52:08.365617 1842 raft_consensus.cc:491] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.365772 1842 raft_consensus.cc:513] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.366067 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet abd6f9121a9d4b39adde1b880a875e23 (table default.Table_5d5a62a1608c45929c8fa1c853a06a27 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]) successfully deleted
I20250625 13:52:08.366766 1166 tablet_bootstrap.cc:654] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.366727 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fdca99b734448118e0c42227c78e58f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.368729 1168 ts_tablet_manager.cc:1918] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:08.368837 1168 log.cc:1199] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/b1acc41a93f6433588727811903921fe
I20250625 13:52:08.369206 1168 ts_tablet_manager.cc:1939] T b1acc41a93f6433588727811903921fe P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.370085 1165 ts_tablet_manager.cc:1905] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.371966 1710 ts_tablet_manager.cc:1905] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.379325 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 671) took 2855 ms. Trace:
I20250625 13:52:08.379416 1168 rpcz_store.cc:276] 0625 13:52:05.523893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.525775 (+ 1882us) service_pool.cc:225] Handling call
0625 13:52:08.338706 (+2812931us) ts_tablet_manager.cc:1145] Deleting tablet b1acc41a93f6433588727811903921fe
0625 13:52:08.338717 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.353179 (+ 14462us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.368623 (+ 15444us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.379277 (+ 10654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2812107}
I20250625 13:52:08.379783 1168 tablet_replica.cc:331] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:08.379887 849 leader_election.cc:343] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.379976 1168 raft_consensus.cc:2241] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.380115 1168 raft_consensus.cc:2270] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.380928 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c4265b6c17334e1fa2297818e6d2558a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.381048 851 leader_election.cc:343] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8fdca99b734448118e0c42227c78e58f
I20250625 13:52:08.381168 851 leader_election.cc:304] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:08.381556 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4265b6c17334e1fa2297818e6d2558a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.381867 1837 raft_consensus.cc:2747] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.382191 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet b1acc41a93f6433588727811903921fe (table default.Table_065fd6832b3b458cbedf3713e9d234b3 [id=380637ad0dc14b208e6b6840ded2d5a1]) successfully deleted
I20250625 13:52:08.384696 1842 leader_election.cc:290] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.385607 743 tablet_service.cc:1468] Processing CreateTablet for tablet 753e4767d7b84515ad2c93437e46c65b (DEFAULT_TABLE table=default.Table_3a79d0adb2204bcaa209ad1de0fa64fc [id=ad7755e4d3614d179c349c6c3ceb9d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.386073 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753e4767d7b84515ad2c93437e46c65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.386265 1168 ts_tablet_manager.cc:1905] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.388757 888 tablet_service.cc:1468] Processing CreateTablet for tablet 753e4767d7b84515ad2c93437e46c65b (DEFAULT_TABLE table=default.Table_3a79d0adb2204bcaa209ad1de0fa64fc [id=ad7755e4d3614d179c349c6c3ceb9d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.389196 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753e4767d7b84515ad2c93437e46c65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.399981 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 28f180cd0a3141cc8c0889d7a8ca1d72 (DEFAULT_TABLE table=default.Table_4ecc6555bb17447bb819b95c6b928c7b [id=2d6c8e320f1243b88a9d1a954f5d4804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.400454 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f180cd0a3141cc8c0889d7a8ca1d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.400414 1710 ts_tablet_manager.cc:1918] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.403507 1710 log.cc:1199] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/455884b62ee44b6da53543093f721027
I20250625 13:52:08.404217 1166 tablet_bootstrap.cc:492] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.404367 1166 ts_tablet_manager.cc:1397] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.061s user 0.014s sys 0.006s
I20250625 13:52:08.405139 1166 raft_consensus.cc:357] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.405318 1166 raft_consensus.cc:383] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.405366 1166 raft_consensus.cc:738] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.405613 1166 consensus_queue.cc:260] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.407025 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4190488f5e9f436d986974482c2e5ecd" } new_table_name: "default.Table_9545d17bca8440d5894a77ccd3fafc1b" modify_external_catalogs: true
I20250625 13:52:08.408756 1710 ts_tablet_manager.cc:1939] T 455884b62ee44b6da53543093f721027 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:08.408857 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
I20250625 13:52:08.410332 1710 tablet_replica.cc:331] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:08.410476 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 455884b62ee44b6da53543093f721027 (table default.Table_f51c5c5fe5394b598404b1944ee7d0b5 [id=67c521d398af4fc4bfe9e6c23d400906]) successfully deleted
I20250625 13:52:08.410557 1710 raft_consensus.cc:2241] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.410751 1710 raft_consensus.cc:2270] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.411392 1166 ts_tablet_manager.cc:1428] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:08.412444 1168 ts_tablet_manager.cc:1918] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.412581 1168 log.cc:1199] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/19017cc372af43b2b5074a8990375ede
I20250625 13:52:08.414436 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
I20250625 13:52:08.415659 1166 tablet_bootstrap.cc:492] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.412873 1168 ts_tablet_manager.cc:1939] T 19017cc372af43b2b5074a8990375ede P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.417166 752 tablet_service.cc:1468] Processing CreateTablet for tablet 28f180cd0a3141cc8c0889d7a8ca1d72 (DEFAULT_TABLE table=default.Table_4ecc6555bb17447bb819b95c6b928c7b [id=2d6c8e320f1243b88a9d1a954f5d4804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.417344 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 669) took 2893 ms. Trace:
I20250625 13:52:08.417515 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f180cd0a3141cc8c0889d7a8ca1d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.417421 1168 rpcz_store.cc:276] 0625 13:52:05.523596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.531183 (+ 7587us) service_pool.cc:225] Handling call
0625 13:52:08.379731 (+2848548us) ts_tablet_manager.cc:1145] Deleting tablet 19017cc372af43b2b5074a8990375ede
0625 13:52:08.379742 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.389149 (+ 9407us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.412325 (+ 23176us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.417323 (+ 4998us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2848331}
I20250625 13:52:08.417753 1168 tablet_replica.cc:331] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.418996 1166 tablet_bootstrap.cc:654] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.420339 1168 raft_consensus.cc:2241] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.420523 1168 raft_consensus.cc:2270] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.421015 1168 ts_tablet_manager.cc:1905] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.426136 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3cb5cc5957974c9580239ecdf22f26cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.426724 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cb5cc5957974c9580239ecdf22f26cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.426746 1168 ts_tablet_manager.cc:1918] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.426836 1168 log.cc:1199] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/f065b4e3218b4e6e8a2ac3a70eede13b
I20250625 13:52:08.427100 1168 ts_tablet_manager.cc:1939] T f065b4e3218b4e6e8a2ac3a70eede13b P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.427878 1171 tablet_bootstrap.cc:492] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.429715 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 19017cc372af43b2b5074a8990375ede (table default.Table_d675ca14ac884a35aab70e66cdcef846 [id=fed9fb5422bc435ab14ed8f178cc95a1]) successfully deleted
I20250625 13:52:08.430405 1171 ts_tablet_manager.cc:1397] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.009s
I20250625 13:52:08.431161 1171 raft_consensus.cc:357] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.431350 1171 raft_consensus.cc:383] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.431412 1171 raft_consensus.cc:738] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.431614 1171 consensus_queue.cc:260] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.432088 1171 ts_tablet_manager.cc:1428] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250625 13:52:08.436703 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 882152a3a5a2458d803ba7f75b31bc30 consensus state has no leader
I20250625 13:52:08.437273 1171 tablet_bootstrap.cc:492] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.439174 1171 tablet_bootstrap.cc:654] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.439446 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:08.442211 1710 ts_tablet_manager.cc:1905] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.445041 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:08.445647 892 tablet_service.cc:1468] Processing CreateTablet for tablet 28f180cd0a3141cc8c0889d7a8ca1d72 (DEFAULT_TABLE table=default.Table_4ecc6555bb17447bb819b95c6b928c7b [id=2d6c8e320f1243b88a9d1a954f5d4804]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.448634 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1b3d9a83b5544828a58cdd409a9417e7" } new_table_name: "default.Table_a73ed5341bcb402dba08bfa062ead863" modify_external_catalogs: true
I20250625 13:52:08.447150 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 676) took 2849 ms. Trace:
W20250625 13:52:08.454468 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbb6967bc38b42f7931f6a6009603d8f consensus state has no leader
I20250625 13:52:08.456023 751 tablet_service.cc:1468] Processing CreateTablet for tablet b493f32299ba42159c594849e0fe39c4 (DEFAULT_TABLE table=default.Table_cc87fc1ee177485192b77fe418e55945 [id=396b52a757ec48d4b185c395712081ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.456064 881 tablet_service.cc:1468] Processing CreateTablet for tablet b493f32299ba42159c594849e0fe39c4 (DEFAULT_TABLE table=default.Table_cc87fc1ee177485192b77fe418e55945 [id=396b52a757ec48d4b185c395712081ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.456023 746 tablet_service.cc:1468] Processing CreateTablet for tablet da6bcb2055b94749879607559e62f6eb (DEFAULT_TABLE table=default.Table_4187a17a1e4245cf8d7d644bb7713836 [id=bd68ead638054b7182148583564c3b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.457285 1015 tablet_service.cc:1468] Processing CreateTablet for tablet da6bcb2055b94749879607559e62f6eb (DEFAULT_TABLE table=default.Table_4187a17a1e4245cf8d7d644bb7713836 [id=bd68ead638054b7182148583564c3b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.457687 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da6bcb2055b94749879607559e62f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.460163 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f180cd0a3141cc8c0889d7a8ca1d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.462330 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b493f32299ba42159c594849e0fe39c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.463913 1166 tablet_bootstrap.cc:492] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.464041 1166 ts_tablet_manager.cc:1397] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250625 13:52:08.465435 1166 raft_consensus.cc:357] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.465595 1166 raft_consensus.cc:383] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.465638 1166 raft_consensus.cc:738] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.465883 1166 consensus_queue.cc:260] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.466607 1166 ts_tablet_manager.cc:1428] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:08.468057 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6fc366fe7a4f4086a2f2cf60731c8089"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.468659 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fc366fe7a4f4086a2f2cf60731c8089 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.473587 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da6bcb2055b94749879607559e62f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.475291 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 063a793e0d7747668d2728fdc2160b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.475929 881 tablet_service.cc:1468] Processing CreateTablet for tablet da6bcb2055b94749879607559e62f6eb (DEFAULT_TABLE table=default.Table_4187a17a1e4245cf8d7d644bb7713836 [id=bd68ead638054b7182148583564c3b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.476274 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da6bcb2055b94749879607559e62f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.454337 1168 rpcz_store.cc:276] 0625 13:52:05.598185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.598244 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:08.417711 (+2819467us) ts_tablet_manager.cc:1145] Deleting tablet f065b4e3218b4e6e8a2ac3a70eede13b
0625 13:52:08.417720 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.423932 (+ 6212us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.426668 (+ 2736us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.447128 (+ 20460us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2819267}
I20250625 13:52:08.479809 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet f065b4e3218b4e6e8a2ac3a70eede13b (table default.Table_71166441017840fa9a67752209689ee2 [id=fc9bcf7791ec4c069f0bcae18943cac9]) successfully deleted
I20250625 13:52:08.479868 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_013cf6cca16c4490ad8457353b68afc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.480129 1168 tablet_replica.cc:331] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:08.480413 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_013cf6cca16c4490ad8457353b68afc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.480327 1168 raft_consensus.cc:2241] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.480697 1168 raft_consensus.cc:2270] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.481206 1168 ts_tablet_manager.cc:1905] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.482089 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_07eaa93fdfa1484f98c567581341f0e9" } new_table_name: "default.Table_123acc43ac8c41d4919dcf68f4e92794" modify_external_catalogs: true
I20250625 13:52:08.483160 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1882ec53ba0a4451a9fcf2b494242aea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.483434 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_722e0f5afe094da88dd5c6985558f1c1" } modify_external_catalogs: true
I20250625 13:52:08.483644 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_722e0f5afe094da88dd5c6985558f1c1" } modify_external_catalogs: true
W20250625 13:52:08.483747 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1882ec53ba0a4451a9fcf2b494242aea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.485129 1166 tablet_bootstrap.cc:492] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.486361 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6277ec7df0c84c3b9af326711152e888" } new_table_name: "default.Table_34d8b1f4040c4fc08b1f847c31060596" modify_external_catalogs: true
I20250625 13:52:08.487255 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9798c064000341b5bc384b0dee1787d3" } new_table_name: "default.Table_f26987562c25452388d457eb3b600595" modify_external_catalogs: true
I20250625 13:52:08.487710 1166 tablet_bootstrap.cc:654] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.491170 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b493f32299ba42159c594849e0fe39c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.493419 1171 tablet_bootstrap.cc:492] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.493552 1171 ts_tablet_manager.cc:1397] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.004s
I20250625 13:52:08.494431 1171 raft_consensus.cc:357] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.494599 1171 raft_consensus.cc:383] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.494671 1171 raft_consensus.cc:738] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.494876 1171 consensus_queue.cc:260] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.496068 1171 ts_tablet_manager.cc:1428] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:08.496424 1171 tablet_bootstrap.cc:492] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.498215 1171 tablet_bootstrap.cc:654] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.500228 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet dc1c9cea3d5a405a958c6c7585dbef39
W20250625 13:52:08.501040 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a525544e77b4a96a79c8876c48bfae1 consensus state has no leader
W20250625 13:52:08.502204 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb9fcf91fa304a359213a3e5267a9cac has no consensus state
I20250625 13:52:08.502245 746 tablet_service.cc:1515] Processing DeleteTablet for tablet dc1c9cea3d5a405a958c6c7585dbef39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.502723 1170 tablet_bootstrap.cc:492] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.502914 1170 ts_tablet_manager.cc:1397] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.254s user 0.022s sys 0.024s
I20250625 13:52:08.503733 1170 raft_consensus.cc:357] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.504120 746 tablet_service.cc:1468] Processing CreateTablet for tablet 512556b0f53f41f7a9eae35997911ec4 (DEFAULT_TABLE table=default.Table_ab516897fdf84efa9c63d6f43223d516 [id=7d41ca4569a04ce4a394c4b7a5d7dca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.504274 1170 raft_consensus.cc:383] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.504341 1170 raft_consensus.cc:738] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.504527 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512556b0f53f41f7a9eae35997911ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.505008 1170 consensus_queue.cc:260] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.509258 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.511348 1842 raft_consensus.cc:491] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.511480 1842 raft_consensus.cc:513] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.512075 1842 leader_election.cc:290] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.512846 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc75a0a8421b457089cc707d9eb6b76f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.513062 1045 raft_consensus.cc:2466] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.513546 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 512556b0f53f41f7a9eae35997911ec4 (DEFAULT_TABLE table=default.Table_ab516897fdf84efa9c63d6f43223d516 [id=7d41ca4569a04ce4a394c4b7a5d7dca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.513672 849 leader_election.cc:304] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.513911 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512556b0f53f41f7a9eae35997911ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.513972 1837 raft_consensus.cc:491] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.514079 1837 raft_consensus.cc:513] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.514642 1837 leader_election.cc:290] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.514773 1837 raft_consensus.cc:2802] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.514837 1837 raft_consensus.cc:491] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.514895 1837 raft_consensus.cc:3058] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.515364 1166 tablet_bootstrap.cc:492] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.515487 1166 ts_tablet_manager.cc:1397] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20250625 13:52:08.516199 1166 raft_consensus.cc:357] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.516341 1166 raft_consensus.cc:383] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.516382 1166 raft_consensus.cc:738] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.516551 1166 consensus_queue.cc:260] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.518028 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e529f5bf13f4ee69e66402b1ee252d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.518198 1045 raft_consensus.cc:2466] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.518620 849 leader_election.cc:304] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.518851 1748 raft_consensus.cc:2802] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.518923 1748 raft_consensus.cc:491] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.518988 1748 raft_consensus.cc:3058] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.520640 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e77736191544e8da32ca9927f0a5804 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.521790 1748 raft_consensus.cc:513] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.522758 1748 leader_election.cc:290] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.523388 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e529f5bf13f4ee69e66402b1ee252d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:08.523551 1045 raft_consensus.cc:3058] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.526543 1045 raft_consensus.cc:2466] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.527490 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc75a0a8421b457089cc707d9eb6b76f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:08.527712 774 raft_consensus.cc:2466] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.527768 1018 tablet_service.cc:1468] Processing CreateTablet for tablet b493f32299ba42159c594849e0fe39c4 (DEFAULT_TABLE table=default.Table_cc87fc1ee177485192b77fe418e55945 [id=396b52a757ec48d4b185c395712081ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.528185 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b493f32299ba42159c594849e0fe39c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.528152 849 leader_election.cc:304] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.528537 1842 raft_consensus.cc:2802] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.528645 1842 raft_consensus.cc:695] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:08.528863 1842 consensus_queue.cc:237] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.530423 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e529f5bf13f4ee69e66402b1ee252d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:08.530609 777 raft_consensus.cc:3058] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.531778 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e529f5bf13f4ee69e66402b1ee252d4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:08.531868 1170 ts_tablet_manager.cc:1428] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250625 13:52:08.532781 1171 tablet_bootstrap.cc:492] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.532925 1171 ts_tablet_manager.cc:1397] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20250625 13:52:08.533136 889 tablet_service.cc:1468] Processing CreateTablet for tablet 512556b0f53f41f7a9eae35997911ec4 (DEFAULT_TABLE table=default.Table_ab516897fdf84efa9c63d6f43223d516 [id=7d41ca4569a04ce4a394c4b7a5d7dca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.533494 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512556b0f53f41f7a9eae35997911ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.533674 1171 raft_consensus.cc:357] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.533810 1171 raft_consensus.cc:383] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.533859 1171 raft_consensus.cc:738] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.534077 1171 consensus_queue.cc:260] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.534188 777 raft_consensus.cc:2466] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.537580 1837 raft_consensus.cc:513] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.538976 1837 leader_election.cc:290] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:08.539598 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 1a525544e77b4a96a79c8876c48bfae1 (table default.Table_123acc43ac8c41d4919dcf68f4e92794 [id=6980d122c3da49cdac9225d059385366], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1a525544e77b4a96a79c8876c48bfae1 consensus state has no leader
I20250625 13:52:08.540239 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_30f7d431b5314e83b313365e3b0bffd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.540715 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d8668c84f52c4e7299ffffb2c41a9929" } modify_external_catalogs: true
W20250625 13:52:08.540884 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30f7d431b5314e83b313365e3b0bffd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.541254 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1b6dba3c0b1f4e9da6c17ca8017e4b93" } modify_external_catalogs: true
I20250625 13:52:08.541368 1170 tablet_bootstrap.cc:492] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.541424 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1b6dba3c0b1f4e9da6c17ca8017e4b93" } modify_external_catalogs: true
I20250625 13:52:08.542171 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e4c0ca74c2f4252a4dc3c7e85afc3d0" } modify_external_catalogs: true
I20250625 13:52:08.542315 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e4c0ca74c2f4252a4dc3c7e85afc3d0" } modify_external_catalogs: true
I20250625 13:52:08.542953 1170 tablet_bootstrap.cc:654] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.543041 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_81c1b426fda24349805e3e5cf9fdd561" } modify_external_catalogs: true
I20250625 13:52:08.543179 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_81c1b426fda24349805e3e5cf9fdd561" } modify_external_catalogs: true
I20250625 13:52:08.545840 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_65d439aaf9c9480f8d1b9a3889e23a7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.546392 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65d439aaf9c9480f8d1b9a3889e23a7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.547405 593 master_service.cc:948] ReplaceTablet: received request to replace tablet c7ad5e90188847f38894fc3c1f8f2693 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:08.548950 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6755f077e9a0460d86f174be4593542a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.549397 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6755f077e9a0460d86f174be4593542a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.550546 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc75a0a8421b457089cc707d9eb6b76f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:08.550719 777 raft_consensus.cc:3058] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.540882 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d8668c84f52c4e7299ffffb2c41a9929" } modify_external_catalogs: true
I20250625 13:52:08.553864 777 raft_consensus.cc:2466] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.554322 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc75a0a8421b457089cc707d9eb6b76f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:08.554701 1045 raft_consensus.cc:3058] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.557626 1045 raft_consensus.cc:2466] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.559267 851 leader_election.cc:304] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.559669 1748 raft_consensus.cc:2802] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.559792 1748 raft_consensus.cc:695] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:08.560012 1748 consensus_queue.cc:237] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.560174 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1d9bd351536c4558b189cebf8b793343
I20250625 13:52:08.561486 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9bd351536c4558b189cebf8b793343 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.562578 881 tablet_service.cc:1468] Processing CreateTablet for tablet 2b29392cfeb14f46a60fd0731f220127 (DEFAULT_TABLE table=default.Table_8b535d0341e94ac2b1ea25b603b076ea [id=c089310a3afc47c0b5b1086bee6d70b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.563045 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b29392cfeb14f46a60fd0731f220127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.566485 741 tablet_service.cc:1468] Processing CreateTablet for tablet c9ae350a9dbf476081dc56a709de42c8 (DEFAULT_TABLE table=default.Table_7f21a5a5b40740a28dd60ad19524d6e4 [id=a9ccf25b783645e3a7e0553e3ed4f8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.567019 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9bd351536c4558b189cebf8b793343 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.567077 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9ae350a9dbf476081dc56a709de42c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.567386 887 tablet_service.cc:1468] Processing CreateTablet for tablet c9ae350a9dbf476081dc56a709de42c8 (DEFAULT_TABLE table=default.Table_7f21a5a5b40740a28dd60ad19524d6e4 [id=a9ccf25b783645e3a7e0553e3ed4f8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.567747 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9ae350a9dbf476081dc56a709de42c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.570029 1171 ts_tablet_manager.cc:1428] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.037s user 0.004s sys 0.001s
I20250625 13:52:08.570305 1171 tablet_bootstrap.cc:492] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.571878 1171 tablet_bootstrap.cc:654] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.574581 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d2d334c69d834f9d89194a4ce4dc2fdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.575229 1168 ts_tablet_manager.cc:1918] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.575349 1168 log.cc:1199] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/11711546f0e14f9a9eabf003f49a69b7
I20250625 13:52:08.575769 1168 ts_tablet_manager.cc:1939] T 11711546f0e14f9a9eabf003f49a69b7 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.576467 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 686) took 2865 ms. Trace:
I20250625 13:52:08.576561 1168 rpcz_store.cc:276] 0625 13:52:05.710937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.710994 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:08.480074 (+2769080us) ts_tablet_manager.cc:1145] Deleting tablet 11711546f0e14f9a9eabf003f49a69b7
0625 13:52:08.480086 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.489640 (+ 9554us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.575139 (+ 85499us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.576448 (+ 1309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2768879}
W20250625 13:52:08.575800 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2d334c69d834f9d89194a4ce4dc2fdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.577913 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 11711546f0e14f9a9eabf003f49a69b7 (table default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0]) successfully deleted
I20250625 13:52:08.576581 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9f3212d0b8de440e82156323ecb40a7d
I20250625 13:52:08.579038 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f3212d0b8de440e82156323ecb40a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.581048 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
W20250625 13:52:08.581590 588 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ab7202a8761444a0a731993d890fcdeb consensus state has no leader
I20250625 13:52:08.581873 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f3212d0b8de440e82156323ecb40a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.582589 1166 ts_tablet_manager.cc:1428] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.067s user 0.001s sys 0.000s
I20250625 13:52:08.583906 2107 raft_consensus.cc:491] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.584033 2107 raft_consensus.cc:513] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.584857 2107 leader_election.cc:290] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:08.585458 1166 tablet_bootstrap.cc:492] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.585492 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86af0ea6648045a1ac4ba903d20849b7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:08.585685 774 raft_consensus.cc:2466] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.586119 851 leader_election.cc:304] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.586726 2107 raft_consensus.cc:2802] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.586812 2107 raft_consensus.cc:491] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.587255 2107 raft_consensus.cc:3058] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.589999 2107 raft_consensus.cc:513] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.590623 591 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:08.590842 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86af0ea6648045a1ac4ba903d20849b7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:08.591002 777 raft_consensus.cc:3058] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.591732 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 024f6cb363594420b0d5480fa79d156b
I20250625 13:52:08.592803 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 024f6cb363594420b0d5480fa79d156b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.594286 777 raft_consensus.cc:2466] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.591756 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_454bbba765254b009050c81e4147cb8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.594817 851 leader_election.cc:304] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:08.594980 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_454bbba765254b009050c81e4147cb8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.596835 757 tablet_service.cc:1468] Processing CreateTablet for tablet f5203c047f514dbcaf57087dd8c3e2b6 (DEFAULT_TABLE table=default.Table_a40662133d63412ea32d88ad295176a6 [id=daebf8dbb2dd4d84b4c21c9a1770ee60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.597263 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5203c047f514dbcaf57087dd8c3e2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.597548 744 tablet_service.cc:1468] Processing CreateTablet for tablet 066db01f791542a38cfb6e408c4836ed (DEFAULT_TABLE table=default.Table_906b9e1a3a8147fb935f7a96968ed294 [id=317ce1c5a5774d6ca8d00ed976bf8615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.597920 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066db01f791542a38cfb6e408c4836ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.599748 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cc354854cd8f4c13b5d9f37728037992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.600270 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc354854cd8f4c13b5d9f37728037992 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.600298 2115 raft_consensus.cc:2802] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.600400 2115 raft_consensus.cc:695] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:08.600833 2115 consensus_queue.cc:237] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.601629 1168 tablet_replica.cc:331] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.601833 1168 raft_consensus.cc:2241] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.601962 1168 raft_consensus.cc:2270] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.602437 1168 ts_tablet_manager.cc:1905] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.605253 887 tablet_service.cc:1468] Processing CreateTablet for tablet f5203c047f514dbcaf57087dd8c3e2b6 (DEFAULT_TABLE table=default.Table_a40662133d63412ea32d88ad295176a6 [id=daebf8dbb2dd4d84b4c21c9a1770ee60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.605654 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5203c047f514dbcaf57087dd8c3e2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.605952 2107 leader_election.cc:290] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:08.606145 2107 raft_consensus.cc:491] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.606253 2107 raft_consensus.cc:513] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.607483 2107 leader_election.cc:290] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.612915 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c7ad5e90188847f38894fc3c1f8f2693
I20250625 13:52:08.613514 593 catalog_manager.cc:6652] ReplaceTablet: tablet c7ad5e90188847f38894fc3c1f8f2693 deleted and replaced by tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9
I20250625 13:52:08.614025 744 tablet_service.cc:1515] Processing DeleteTablet for tablet c7ad5e90188847f38894fc3c1f8f2693 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.615507 881 tablet_service.cc:1515] Processing DeleteTablet for tablet c7ad5e90188847f38894fc3c1f8f2693 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.566490 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9bd351536c4558b189cebf8b793343 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.567598 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 2b29392cfeb14f46a60fd0731f220127 (DEFAULT_TABLE table=default.Table_8b535d0341e94ac2b1ea25b603b076ea [id=c089310a3afc47c0b5b1086bee6d70b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.619953 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b29392cfeb14f46a60fd0731f220127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.578400 1170 tablet_bootstrap.cc:492] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.581362 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f3212d0b8de440e82156323ecb40a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.587008 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86af0ea6648045a1ac4ba903d20849b7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.594352 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 024f6cb363594420b0d5480fa79d156b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.624295 1170 ts_tablet_manager.cc:1397] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.000s
I20250625 13:52:08.625118 1170 raft_consensus.cc:357] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.625298 1170 raft_consensus.cc:383] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.625277 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 81a359c798024b2a99aa3b307cc1967f
I20250625 13:52:08.625347 1170 raft_consensus.cc:738] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.625706 1170 consensus_queue.cc:260] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.626116 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 81a359c798024b2a99aa3b307cc1967f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.627177 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 81a359c798024b2a99aa3b307cc1967f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.629086 1170 ts_tablet_manager.cc:1428] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:52:08.629979 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.630805 1170 tablet_bootstrap.cc:492] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.632448 1170 tablet_bootstrap.cc:654] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:08.633395 849 leader_election.cc:343] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.635710 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 81a359c798024b2a99aa3b307cc1967f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.638732 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_74ae228ed14a49bc95616379e50d5936" } new_table_name: "default.Table_71df0e98edd34e26ab99f41178b13683" modify_external_catalogs: true
I20250625 13:52:08.594609 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet dc1c9cea3d5a405a958c6c7585dbef39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.597568 1023 tablet_service.cc:1468] Processing CreateTablet for tablet c9ae350a9dbf476081dc56a709de42c8 (DEFAULT_TABLE table=default.Table_7f21a5a5b40740a28dd60ad19524d6e4 [id=a9ccf25b783645e3a7e0553e3ed4f8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.644143 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9ae350a9dbf476081dc56a709de42c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.598680 1017 tablet_service.cc:1468] Processing CreateTablet for tablet f5203c047f514dbcaf57087dd8c3e2b6 (DEFAULT_TABLE table=default.Table_a40662133d63412ea32d88ad295176a6 [id=daebf8dbb2dd4d84b4c21c9a1770ee60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.647652 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5203c047f514dbcaf57087dd8c3e2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.648998 1170 tablet_bootstrap.cc:492] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.649120 1170 ts_tablet_manager.cc:1397] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20250625 13:52:08.649868 1170 raft_consensus.cc:357] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.650036 1170 raft_consensus.cc:383] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.650086 1170 raft_consensus.cc:738] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.606030 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86af0ea6648045a1ac4ba903d20849b7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:08.607686 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04dd85668a47461e9e82f8b251468ad3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:08.652138 849 leader_election.cc:343] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:08.652345 849 leader_election.cc:343] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.650275 1170 consensus_queue.cc:260] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.657991 1710 ts_tablet_manager.cc:1918] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.658078 1710 log.cc:1199] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/1effed0fa10d409abcb31dbba54861d6
I20250625 13:52:08.658334 1710 ts_tablet_manager.cc:1939] T 1effed0fa10d409abcb31dbba54861d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:08.659670 1171 tablet_bootstrap.cc:492] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.659828 1171 ts_tablet_manager.cc:1397] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.002s
I20250625 13:52:08.660637 1171 raft_consensus.cc:357] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.660815 1171 raft_consensus.cc:383] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.660868 1171 raft_consensus.cc:738] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.661068 1171 consensus_queue.cc:260] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.661062 757 tablet_service.cc:1468] Processing CreateTablet for tablet 645d19afcf124622b379669e6a452e09 (DEFAULT_TABLE table=default.Table_43b8ed6d4ad341448518408d6dee8613 [id=565e52069ef74e7f81232b7d1dec2cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.661111 1170 ts_tablet_manager.cc:1428] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250625 13:52:08.661626 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645d19afcf124622b379669e6a452e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.661680 1171 ts_tablet_manager.cc:1428] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:08.661947 1171 tablet_bootstrap.cc:492] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.662897 744 tablet_service.cc:1468] Processing CreateTablet for tablet 67e05cc542384b4b8117a1d9330db24b (DEFAULT_TABLE table=default.Table_ac4d598c675a46d1bbd2a08d9652f2c7 [id=b36fcb27ef404ffeaa1532ad1063dbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.615298 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet c7ad5e90188847f38894fc3c1f8f2693 with delete_type TABLET_DATA_DELETED (replaced by tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:08.663230 1837 raft_consensus.cc:491] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.663429 1837 raft_consensus.cc:513] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.663514 1170 tablet_bootstrap.cc:492] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:08.663605 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e05cc542384b4b8117a1d9330db24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.664273 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf73b4ab99c466abcfedf1b200e743b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:08.664732 849 leader_election.cc:343] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.665051 1837 leader_election.cc:290] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.665125 1170 tablet_bootstrap.cc:654] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.665930 1168 ts_tablet_manager.cc:1918] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.666041 1168 log.cc:1199] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/8e29e7b55570483b80c6e7a1f227fa7b
I20250625 13:52:08.666329 1168 ts_tablet_manager.cc:1939] T 8e29e7b55570483b80c6e7a1f227fa7b P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.667188 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 687) took 2956 ms. Trace:
I20250625 13:52:08.667272 1168 rpcz_store.cc:276] 0625 13:52:05.711088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.711139 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:08.599270 (+2888131us) ts_tablet_manager.cc:1145] Deleting tablet 8e29e7b55570483b80c6e7a1f227fa7b
0625 13:52:08.599280 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.609419 (+ 10139us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.654921 (+ 45502us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.667168 (+ 12247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2887810}
I20250625 13:52:08.668008 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 689) took 2924 ms. Trace:
I20250625 13:52:08.668088 1168 rpcz_store.cc:276] 0625 13:52:05.743532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.764036 (+ 20504us) service_pool.cc:225] Handling call
0625 13:52:08.667894 (+2903858us) ts_tablet_manager.cc:1145] Deleting tablet cab4329d58404014833ae5029f8f296d
0625 13:52:08.667903 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.667995 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2903453}
I20250625 13:52:08.668543 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 688) took 2926 ms. Trace:
I20250625 13:52:08.668617 1168 rpcz_store.cc:276] 0625 13:52:05.741986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.764276 (+ 22290us) service_pool.cc:225] Handling call
0625 13:52:08.668402 (+2904126us) ts_tablet_manager.cc:1145] Deleting tablet d8985f259eb54ffdab29a9f6a2c5ae16
0625 13:52:08.668410 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.668520 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2903914}
I20250625 13:52:08.669036 1168 tablet_replica.cc:331] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.669581 741 tablet_service.cc:1468] Processing CreateTablet for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 (DEFAULT_TABLE table=default.Table_b584197acb904919a3681da75860d445 [id=6c74ad3d412f43b08a5ae805f8a0d9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.669854 1168 raft_consensus.cc:2241] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.670213 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.670650 1168 raft_consensus.cc:2270] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.671190 1168 ts_tablet_manager.cc:1905] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.671540 1171 tablet_bootstrap.cc:654] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.656478 1166 tablet_bootstrap.cc:654] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.677358 1166 tablet_bootstrap.cc:492] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.677479 1166 ts_tablet_manager.cc:1397] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20250625 13:52:08.678284 1166 raft_consensus.cc:357] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.678471 1166 raft_consensus.cc:383] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.678534 1166 raft_consensus.cc:738] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.680040 1837 raft_consensus.cc:491] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.680174 1837 raft_consensus.cc:513] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.680969 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 645d19afcf124622b379669e6a452e09 (DEFAULT_TABLE table=default.Table_43b8ed6d4ad341448518408d6dee8613 [id=565e52069ef74e7f81232b7d1dec2cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.681309 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b0037d09796455cbaaadfaa9c010673" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:08.681841 849 leader_election.cc:343] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.681877 887 tablet_service.cc:1468] Processing CreateTablet for tablet 645d19afcf124622b379669e6a452e09 (DEFAULT_TABLE table=default.Table_43b8ed6d4ad341448518408d6dee8613 [id=565e52069ef74e7f81232b7d1dec2cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.682219 1837 leader_election.cc:290] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.682209 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 (DEFAULT_TABLE table=default.Table_b584197acb904919a3681da75860d445 [id=6c74ad3d412f43b08a5ae805f8a0d9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.682821 1170 tablet_bootstrap.cc:492] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:08.682709 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 61542e4ef6a04dbb8a1b989a0767507a (DEFAULT_TABLE table=default.Table_0a8b5ac0e4a447a6b313a397677475fa [id=d00809f4e4e34bf8955876889a8a202d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.682956 1170 ts_tablet_manager.cc:1397] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20250625 13:52:08.682906 893 tablet_service.cc:1468] Processing CreateTablet for tablet 61542e4ef6a04dbb8a1b989a0767507a (DEFAULT_TABLE table=default.Table_0a8b5ac0e4a447a6b313a397677475fa [id=d00809f4e4e34bf8955876889a8a202d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.683377 891 tablet_service.cc:1515] Processing DeleteTablet for tablet dc1c9cea3d5a405a958c6c7585dbef39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.683746 1170 raft_consensus.cc:357] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.683923 1170 raft_consensus.cc:383] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.683884 884 tablet_service.cc:1468] Processing CreateTablet for tablet 67e05cc542384b4b8117a1d9330db24b (DEFAULT_TABLE table=default.Table_ac4d598c675a46d1bbd2a08d9652f2c7 [id=b36fcb27ef404ffeaa1532ad1063dbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.683982 1170 raft_consensus.cc:738] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.682232 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645d19afcf124622b379669e6a452e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.684665 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e05cc542384b4b8117a1d9330db24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.684891 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61542e4ef6a04dbb8a1b989a0767507a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.686724 1170 consensus_queue.cc:260] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.684505 881 tablet_service.cc:1468] Processing CreateTablet for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 (DEFAULT_TABLE table=default.Table_b584197acb904919a3681da75860d445 [id=6c74ad3d412f43b08a5ae805f8a0d9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.691732 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.691955 1170 ts_tablet_manager.cc:1428] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 13:52:08.692292 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 066db01f791542a38cfb6e408c4836ed (DEFAULT_TABLE table=default.Table_906b9e1a3a8147fb935f7a96968ed294 [id=317ce1c5a5774d6ca8d00ed976bf8615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.693718 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 8e29e7b55570483b80c6e7a1f227fa7b (table default.Table_2b3cf346c6724405a0e96eef702fc43e [id=4891101f0201404ab9d5c88c094ce3ff]) successfully deleted
W20250625 13:52:08.694096 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet cab4329d58404014833ae5029f8f296d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cab4329d58404014833ae5029f8f296d already in progress: creating tablet
W20250625 13:52:08.694396 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d8985f259eb54ffdab29a9f6a2c5ae16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8985f259eb54ffdab29a9f6a2c5ae16 already in progress: creating tablet
I20250625 13:52:08.694460 1168 ts_tablet_manager.cc:1918] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.694556 1168 log.cc:1199] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/8b7062c34fd143bf86b743055a27db8c
I20250625 13:52:08.694880 1168 ts_tablet_manager.cc:1939] T 8b7062c34fd143bf86b743055a27db8c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.695080 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e460144bc7964ddba939c069278e887d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.695726 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e460144bc7964ddba939c069278e887d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.696004 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_af34e284a9af4e60bc69bcdbf5acc655"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.696513 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af34e284a9af4e60bc69bcdbf5acc655 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.699522 1170 tablet_bootstrap.cc:492] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:08.699558 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f002689f9655413b9e1954d17fcfe92f has no consensus state
I20250625 13:52:08.699978 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1e889135ee254afca2317cec6787455e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.700456 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e889135ee254afca2317cec6787455e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.701673 1170 tablet_bootstrap.cc:654] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.702229 593 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:08.702991 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_70a5975b997b4664ae947803f73e1a27" } new_table_name: "default.Table_2bea8ee8d6074d3785dc33d166fc3485" modify_external_catalogs: true
I20250625 13:52:08.705785 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4d8e666a80a345f7ba2ad05b84ce5a05" } modify_external_catalogs: true
I20250625 13:52:08.705971 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4d8e666a80a345f7ba2ad05b84ce5a05" } modify_external_catalogs: true
I20250625 13:52:08.705497 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 691) took 2905 ms. Trace:
I20250625 13:52:08.707693 1168 rpcz_store.cc:276] 0625 13:52:05.800159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.800198 (+ 39us) service_pool.cc:225] Handling call
0625 13:52:08.668998 (+2868800us) ts_tablet_manager.cc:1145] Deleting tablet 8b7062c34fd143bf86b743055a27db8c
0625 13:52:08.669006 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.680577 (+ 11571us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.694374 (+ 13797us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.705474 (+ 11100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2868557}
I20250625 13:52:08.708532 1168 tablet_replica.cc:331] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.708737 1168 raft_consensus.cc:2241] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.708889 1168 raft_consensus.cc:2270] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.709335 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:08.709412 1168 ts_tablet_manager.cc:1905] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.712209 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_107ccce5d138433db4fb8351f69e40a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.712795 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_107ccce5d138433db4fb8351f69e40a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.714057 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1402caa3f71f4bf7b40629b140616eeb" } new_table_name: "default.Table_0139be8680a54e53983ff6c86c11d76e" modify_external_catalogs: true
I20250625 13:52:08.715276 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645d19afcf124622b379669e6a452e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.715504 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.715859 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4fcd474fe7fd48c0b7a71988682079c4" } new_table_name: "default.Table_653b1c2c1086473ea503f44cc7e84bcc" modify_external_catalogs: true
I20250625 13:52:08.716288 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61542e4ef6a04dbb8a1b989a0767507a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.716672 1166 consensus_queue.cc:260] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.718513 1171 tablet_bootstrap.cc:492] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.718643 1171 ts_tablet_manager.cc:1397] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20250625 13:52:08.719367 1171 raft_consensus.cc:357] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.719512 1171 raft_consensus.cc:383] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.719555 1171 raft_consensus.cc:738] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.719476 597 catalog_manager.cc:5582] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:08.719924 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:08.720161 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
I20250625 13:52:08.721382 1168 ts_tablet_manager.cc:1918] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.721491 1168 log.cc:1199] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a22ade8cefac4140b08882c311ea5181
I20250625 13:52:08.721792 1168 ts_tablet_manager.cc:1939] T a22ade8cefac4140b08882c311ea5181 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.722486 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 690) took 2922 ms. Trace:
I20250625 13:52:08.722565 1168 rpcz_store.cc:276] 0625 13:52:05.800011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.800465 (+ 454us) service_pool.cc:225] Handling call
0625 13:52:08.708482 (+2908017us) ts_tablet_manager.cc:1145] Deleting tablet a22ade8cefac4140b08882c311ea5181
0625 13:52:08.708493 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.718333 (+ 9840us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.721289 (+ 2956us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.722466 (+ 1177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2907661}
I20250625 13:52:08.723671 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066db01f791542a38cfb6e408c4836ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.729441 1166 ts_tablet_manager.cc:1428] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20250625 13:52:08.729770 1166 tablet_bootstrap.cc:492] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.731357 1166 tablet_bootstrap.cc:654] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.733011 1710 tablet_replica.cc:331] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:08.734288 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 430c3bed360b4f32855a8adb99076608 with delete_type TABLET_DATA_DELETED (replaced by tablet 0f63e9037bd74f1eb39bd85aa820ec35) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.741648 744 tablet_service.cc:1468] Processing CreateTablet for tablet c9d010744e88406a828c6ba7b76eda3e (DEFAULT_TABLE table=default.Table_f5f3d8e484094d71808796036298fd34 [id=5927af29a96f49b0a1b5491ab9579c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.742096 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9d010744e88406a828c6ba7b76eda3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.742120 1168 tablet_replica.cc:331] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.742022 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_65b6f087b85f4a06a9131796bd8a943e" } new_table_name: "default.Table_b1ac14afc057412d92fba0065ebb7a1b" modify_external_catalogs: true
I20250625 13:52:08.742295 1168 raft_consensus.cc:2241] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.742465 1168 raft_consensus.cc:2270] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.745564 1168 ts_tablet_manager.cc:1905] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.749446 1710 raft_consensus.cc:2241] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.749614 1710 raft_consensus.cc:2270] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.762491 892 tablet_service.cc:1468] Processing CreateTablet for tablet 066db01f791542a38cfb6e408c4836ed (DEFAULT_TABLE table=default.Table_906b9e1a3a8147fb935f7a96968ed294 [id=317ce1c5a5774d6ca8d00ed976bf8615]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.763018 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066db01f791542a38cfb6e408c4836ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.764544 751 tablet_service.cc:1468] Processing CreateTablet for tablet d0642b80eda44397878897217a29af65 (DEFAULT_TABLE table=default.Table_31c75178c8ab41378e5a92bb7b2ccb9f [id=686b2da203e648d29a9fae358fb9e431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.765075 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0642b80eda44397878897217a29af65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.767762 743 tablet_service.cc:1468] Processing CreateTablet for tablet 61542e4ef6a04dbb8a1b989a0767507a (DEFAULT_TABLE table=default.Table_0a8b5ac0e4a447a6b313a397677475fa [id=d00809f4e4e34bf8955876889a8a202d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:08.768182 581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 73789.5us
I20250625 13:52:08.769289 1168 ts_tablet_manager.cc:1918] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.769430 1168 log.cc:1199] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/c666ff2d6b9b4ed4a18e370826c143fc
I20250625 13:52:08.769797 1168 ts_tablet_manager.cc:1939] T c666ff2d6b9b4ed4a18e370826c143fc P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.770013 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61542e4ef6a04dbb8a1b989a0767507a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.554863 746 tablet_service.cc:1468] Processing CreateTablet for tablet 2b29392cfeb14f46a60fd0731f220127 (DEFAULT_TABLE table=default.Table_8b535d0341e94ac2b1ea25b603b076ea [id=c089310a3afc47c0b5b1086bee6d70b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.557591 1165 ts_tablet_manager.cc:1918] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.771063 1165 log.cc:1199] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/38a23b08a03b48eaa051f670eb7dac65
I20250625 13:52:08.771538 1171 consensus_queue.cc:260] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.772887 1710 ts_tablet_manager.cc:1905] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.772905 756 tablet_service.cc:1468] Processing CreateTablet for tablet 45079deff84a41b488ccdde2f5059431 (DEFAULT_TABLE table=default.Table_9b61484dc29b44ac9d0afd92e1da0ef3 [id=6cc418bddb944dc2b6d6a63452b6bfc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.773236 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45079deff84a41b488ccdde2f5059431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.773407 1165 ts_tablet_manager.cc:1939] T 38a23b08a03b48eaa051f670eb7dac65 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:08.774482 1165 tablet_replica.cc:331] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:08.774704 1165 raft_consensus.cc:2241] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.774890 1165 raft_consensus.cc:2270] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.775521 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 38a23b08a03b48eaa051f670eb7dac65 (table default.Table_91567a207fcd4540937c0e047ea86de5 [id=5fcd0958c61942289b4eb28c077ceec3]) successfully deleted
I20250625 13:52:08.776628 1165 ts_tablet_manager.cc:1905] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.780141 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b29392cfeb14f46a60fd0731f220127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.780423 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 024f6cb363594420b0d5480fa79d156b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.723670 1842 raft_consensus.cc:491] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.780752 1842 raft_consensus.cc:513] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.781764 1842 leader_election.cc:290] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.782087 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "067272869c5547648c90f83a34a036aa" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:08.782541 849 leader_election.cc:343] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.783557 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 8b7062c34fd143bf86b743055a27db8c (table default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]) successfully deleted
I20250625 13:52:08.783807 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet a22ade8cefac4140b08882c311ea5181 (table default.Table_9dfb8c48d5144c808567a16a30cc199e [id=f55ecdcac2234415a085be00cacad571]) successfully deleted
I20250625 13:52:08.785287 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d0642b80eda44397878897217a29af65 (DEFAULT_TABLE table=default.Table_31c75178c8ab41378e5a92bb7b2ccb9f [id=686b2da203e648d29a9fae358fb9e431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.785674 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0642b80eda44397878897217a29af65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.795792 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 692) took 2995 ms. Trace:
I20250625 13:52:08.795907 1168 rpcz_store.cc:276] 0625 13:52:05.800562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.800693 (+ 131us) service_pool.cc:225] Handling call
0625 13:52:08.742066 (+2941373us) ts_tablet_manager.cc:1145] Deleting tablet c666ff2d6b9b4ed4a18e370826c143fc
0625 13:52:08.742076 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.748596 (+ 6520us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.769165 (+ 20569us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.795766 (+ 26601us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15297,"tablet-delete.queue_time_us":2941142}
I20250625 13:52:08.796238 1168 tablet_replica.cc:331] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.796401 1168 raft_consensus.cc:2241] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.796525 1168 raft_consensus.cc:2270] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.796998 1168 ts_tablet_manager.cc:1905] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.796092 1166 tablet_bootstrap.cc:492] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.799456 1166 ts_tablet_manager.cc:1397] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250625 13:52:08.801759 1166 raft_consensus.cc:357] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.801967 1166 raft_consensus.cc:383] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.802026 1166 raft_consensus.cc:738] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.802289 1166 consensus_queue.cc:260] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.806535 2132 tablet.cc:1722] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:08.807334 2133 tablet.cc:1722] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:08.809805 1166 ts_tablet_manager.cc:1428] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 13:52:08.813829 1171 ts_tablet_manager.cc:1428] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.095s user 0.002s sys 0.000s
I20250625 13:52:08.814453 1166 tablet_bootstrap.cc:492] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.816056 1166 tablet_bootstrap.cc:654] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:08.817147 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 222344us
I20250625 13:52:08.817940 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04dd85668a47461e9e82f8b251468ad3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:08.818123 774 raft_consensus.cc:2466] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:08.818581 1171 tablet_bootstrap.cc:492] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.820709 1171 tablet_bootstrap.cc:654] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.821538 1165 ts_tablet_manager.cc:1918] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.821955 1165 log.cc:1199] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/063a793e0d7747668d2728fdc2160b5e
I20250625 13:52:08.822942 746 tablet_service.cc:1468] Processing CreateTablet for tablet 93a0b94879874446b569b3cb3250b185 (DEFAULT_TABLE table=default.Table_53926d5babb045a4854877f904b639ec [id=325dd21568534ee58900ef0b8d79fae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.827647 757 tablet_service.cc:1468] Processing CreateTablet for tablet 53dda116b9ee425e9ce6181e217d53c4 (DEFAULT_TABLE table=default.Table_64adeba774054ea5a73be69a92bf079e [id=70185296dee54d82b614cd879436adbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.828230 1165 ts_tablet_manager.cc:1939] T 063a793e0d7747668d2728fdc2160b5e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:08.829214 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 063a793e0d7747668d2728fdc2160b5e (table default.Table_9c6d8799477b4081b2652a4df3b13762 [id=3398f6ade26340dcae5b0da2eb58034c]) successfully deleted
W20250625 13:52:08.830025 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet dc1c9cea3d5a405a958c6c7585dbef39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc1c9cea3d5a405a958c6c7585dbef39 already in progress: creating tablet
W20250625 13:52:08.830724 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1d9bd351536c4558b189cebf8b793343 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d9bd351536c4558b189cebf8b793343 already in progress: creating tablet
W20250625 13:52:08.831377 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9f3212d0b8de440e82156323ecb40a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f3212d0b8de440e82156323ecb40a7d already in progress: creating tablet
W20250625 13:52:08.831323 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 54964.2us
W20250625 13:52:08.832120 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 024f6cb363594420b0d5480fa79d156b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 024f6cb363594420b0d5480fa79d156b already in progress: creating tablet
W20250625 13:52:08.832834 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c7ad5e90188847f38894fc3c1f8f2693 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7ad5e90188847f38894fc3c1f8f2693 already in progress: creating tablet
W20250625 13:52:08.833659 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 81a359c798024b2a99aa3b307cc1967f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81a359c798024b2a99aa3b307cc1967f already in progress: creating tablet
I20250625 13:52:08.834122 1165 tablet_replica.cc:331] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:08.834563 1165 raft_consensus.cc:2241] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.838166 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a0b94879874446b569b3cb3250b185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.838248 744 tablet_service.cc:1468] Processing CreateTablet for tablet c27caba6c4d14d51b10bf04f131c6a3c (DEFAULT_TABLE table=default.Table_8190e518ba344ab9b9081933923dae13 [id=1e19c242b306485dbce55fdc99fe5720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.840083 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27caba6c4d14d51b10bf04f131c6a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.843493 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53dda116b9ee425e9ce6181e217d53c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.856374 1842 raft_consensus.cc:491] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.856527 1842 raft_consensus.cc:513] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.857060 1842 leader_election.cc:290] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.860718 1171 tablet_bootstrap.cc:492] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.860895 1171 ts_tablet_manager.cc:1397] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20250625 13:52:08.861919 851 leader_election.cc:304] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:08.862594 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d43ba17ebfec4244894d1b6e23200c33" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.863636 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 67e05cc542384b4b8117a1d9330db24b (DEFAULT_TABLE table=default.Table_ac4d598c675a46d1bbd2a08d9652f2c7 [id=b36fcb27ef404ffeaa1532ad1063dbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.864122 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e05cc542384b4b8117a1d9330db24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.866994 1166 tablet_bootstrap.cc:492] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.867830 1166 ts_tablet_manager.cc:1397] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.003s
I20250625 13:52:08.868873 1166 raft_consensus.cc:357] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.870121 1166 raft_consensus.cc:383] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.870327 1166 raft_consensus.cc:738] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:08.863181 849 leader_election.cc:343] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.870733 1166 consensus_queue.cc:260] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.872128 1046 raft_consensus.cc:3058] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.872655 1837 raft_consensus.cc:2802] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.872740 1837 raft_consensus.cc:491] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.872957 1837 raft_consensus.cc:3058] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:08.873648 851 leader_election.cc:343] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 5cf73b4ab99c466abcfedf1b200e743b
I20250625 13:52:08.873776 851 leader_election.cc:304] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.873999 1842 raft_consensus.cc:2747] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:08.874678 851 leader_election.cc:343] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3b0037d09796455cbaaadfaa9c010673
I20250625 13:52:08.874872 851 leader_election.cc:304] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.875129 1842 raft_consensus.cc:2747] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.877743 888 tablet_service.cc:1468] Processing CreateTablet for tablet d0642b80eda44397878897217a29af65 (DEFAULT_TABLE table=default.Table_31c75178c8ab41378e5a92bb7b2ccb9f [id=686b2da203e648d29a9fae358fb9e431]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.878531 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 1effed0fa10d409abcb31dbba54861d6 (table default.Table_ec42501f607c41ceb9d35f3dd2a38f34 [id=d2c6242f48434d3bb6c2dc01fa095a67]) successfully deleted
I20250625 13:52:08.880764 1166 ts_tablet_manager.cc:1428] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 13:52:08.880801 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0642b80eda44397878897217a29af65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.881232 1019 tablet_service.cc:1468] Processing CreateTablet for tablet c9d010744e88406a828c6ba7b76eda3e (DEFAULT_TABLE table=default.Table_f5f3d8e484094d71808796036298fd34 [id=5927af29a96f49b0a1b5491ab9579c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.881831 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 45079deff84a41b488ccdde2f5059431 (DEFAULT_TABLE table=default.Table_9b61484dc29b44ac9d0afd92e1da0ef3 [id=6cc418bddb944dc2b6d6a63452b6bfc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.881969 887 tablet_service.cc:1468] Processing CreateTablet for tablet c9d010744e88406a828c6ba7b76eda3e (DEFAULT_TABLE table=default.Table_f5f3d8e484094d71808796036298fd34 [id=5927af29a96f49b0a1b5491ab9579c45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.882359 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9d010744e88406a828c6ba7b76eda3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.882444 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9d010744e88406a828c6ba7b76eda3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.883829 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45079deff84a41b488ccdde2f5059431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.888631 1837 raft_consensus.cc:513] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.889544 1837 leader_election.cc:290] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:08.890465 1171 raft_consensus.cc:357] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.890733 1171 raft_consensus.cc:383] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.890846 1171 raft_consensus.cc:738] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.891156 1171 consensus_queue.cc:260] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.891842 1171 ts_tablet_manager.cc:1428] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20250625 13:52:08.892594 1171 tablet_bootstrap.cc:492] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.893059 2018 raft_consensus.cc:491] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:08.893189 2018 raft_consensus.cc:513] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.893775 2018 leader_election.cc:290] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:08.895231 851 leader_election.cc:343] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d43ba17ebfec4244894d1b6e23200c33
I20250625 13:52:08.895356 851 leader_election.cc:304] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.895541 1842 raft_consensus.cc:2747] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.895666 1171 tablet_bootstrap.cc:654] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.896015 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:08.896167 912 raft_consensus.cc:2466] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:08.896533 711 leader_election.cc:304] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.896767 1842 consensus_queue.cc:1035] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:08.896847 2018 raft_consensus.cc:2802] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:08.896922 2018 raft_consensus.cc:491] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:08.896983 2018 raft_consensus.cc:3058] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.897106 1168 ts_tablet_manager.cc:1918] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.897193 1168 log.cc:1199] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/58aa3c21191e4bea9c5395d6e32b8717
W20250625 13:52:08.897356 851 leader_election.cc:343] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 067272869c5547648c90f83a34a036aa
I20250625 13:52:08.897439 851 leader_election.cc:304] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:08.897505 1168 ts_tablet_manager.cc:1939] T 58aa3c21191e4bea9c5395d6e32b8717 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.897629 1842 raft_consensus.cc:2747] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:08.898138 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 693) took 3097 ms. Trace:
I20250625 13:52:08.898206 1168 rpcz_store.cc:276] 0625 13:52:05.800972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.801010 (+ 38us) service_pool.cc:225] Handling call
0625 13:52:08.796192 (+2995182us) ts_tablet_manager.cc:1145] Deleting tablet 58aa3c21191e4bea9c5395d6e32b8717
0625 13:52:08.796202 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.838810 (+ 42608us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.897009 (+ 58199us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.898121 (+ 1112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2995018}
I20250625 13:52:08.898480 1168 tablet_replica.cc:331] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:08.898648 1168 raft_consensus.cc:2241] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.898764 1168 raft_consensus.cc:2270] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.899199 1168 ts_tablet_manager.cc:1905] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.900105 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04dd85668a47461e9e82f8b251468ad3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:08.900278 1047 raft_consensus.cc:3058] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.901650 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 2404e712e18146b68757bd77362c1358 with delete_type TABLET_DATA_DELETED (replaced by tablet 68831b9ad1c14f5e9c954cbd06094896) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.902909 1047 raft_consensus.cc:2466] T 04dd85668a47461e9e82f8b251468ad3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:08.903188 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:08.906219 849 leader_election.cc:304] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.906628 1842 raft_consensus.cc:2802] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.906734 1842 raft_consensus.cc:695] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:08.906947 1842 consensus_queue.cc:237] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.908268 2018 raft_consensus.cc:513] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.909423 2018 leader_election.cc:290] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:08.911410 2115 consensus_queue.cc:1035] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:08.911619 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:08.911774 912 raft_consensus.cc:3058] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.885700 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fe2ddd54af1b4192a1378dca82cfcca5" } modify_external_catalogs: true
I20250625 13:52:08.914459 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fe2ddd54af1b4192a1378dca82cfcca5" } modify_external_catalogs: true
I20250625 13:52:08.890172 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9aadc67058554e87bb58d14b911333c4
I20250625 13:52:08.914956 912 raft_consensus.cc:2466] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:08.915587 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bbd45439068f4c9188192e1b98a84784" } modify_external_catalogs: true
I20250625 13:52:08.915742 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bbd45439068f4c9188192e1b98a84784" } modify_external_catalogs: true
W20250625 13:52:08.891084 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4181bbb4af9c4141b587968cb7ef8251 has no consensus state
I20250625 13:52:08.916327 590 master_service.cc:948] ReplaceTablet: received request to replace tablet c9a82adc5aa045f49e092dababafe4f8 from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:08.916832 711 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.917294 711 leader_election.cc:304] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:08.891592 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6e03f752b3f2463b84f66809236ba398"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:08.917600 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:08.917814 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e03f752b3f2463b84f66809236ba398 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.918047 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04dd85668a47461e9e82f8b251468ad3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
W20250625 13:52:08.918170 711 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:08.918190 777 raft_consensus.cc:3058] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:08.920028 1710 ts_tablet_manager.cc:1918] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.920135 1710 log.cc:1199] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/0e77736191544e8da32ca9927f0a5804
I20250625 13:52:08.921077 777 raft_consensus.cc:2466] T 04dd85668a47461e9e82f8b251468ad3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
W20250625 13:52:08.921139 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c5b53c5d2514601b7bcf472ce2d1d84 consensus state has no leader
I20250625 13:52:08.921466 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2a88884ec2db4ae8bc4e94280d4c5022"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.921975 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a88884ec2db4ae8bc4e94280d4c5022 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:08.923137 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a4adf5c25a04a41895e9c35f57e94b2 consensus state has no leader
I20250625 13:52:08.923481 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2fb3cd371d784e4c980b6f63c84d2f09" } modify_external_catalogs: true
I20250625 13:52:08.923635 2018 raft_consensus.cc:2802] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:08.923645 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2fb3cd371d784e4c980b6f63c84d2f09" } modify_external_catalogs: true
I20250625 13:52:08.925419 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff33e931cb1c4e44a12f7dcb8ddad00c" } modify_external_catalogs: true
I20250625 13:52:08.925652 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff33e931cb1c4e44a12f7dcb8ddad00c" } modify_external_catalogs: true
W20250625 13:52:08.927410 581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 48908.5us
W20250625 13:52:08.927659 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 066db01f791542a38cfb6e408c4836ed because the tablet was not found. No further retry: Not found: Tablet not found: 066db01f791542a38cfb6e408c4836ed
I20250625 13:52:08.927678 1710 ts_tablet_manager.cc:1939] T 0e77736191544e8da32ca9927f0a5804 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:08.927860 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet da6bcb2055b94749879607559e62f6eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da6bcb2055b94749879607559e62f6eb already in progress: creating tablet
W20250625 13:52:08.928041 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5cf73b4ab99c466abcfedf1b200e743b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cf73b4ab99c466abcfedf1b200e743b already in progress: creating tablet
W20250625 13:52:08.928207 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c0f8e222d6aa4f5582688f874ce04f4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f8e222d6aa4f5582688f874ce04f4f already in progress: creating tablet
I20250625 13:52:08.929817 1165 raft_consensus.cc:2270] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.931746 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_610e4cc494e244878a60d2382727f85c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.932559 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_610e4cc494e244878a60d2382727f85c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.935438 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 9aadc67058554e87bb58d14b911333c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.935948 1165 ts_tablet_manager.cc:1905] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.937846 1710 tablet_replica.cc:331] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:08.938205 1710 raft_consensus.cc:2241] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:08.936053 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 024f6cb363594420b0d5480fa79d156b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.939814 1171 tablet_bootstrap.cc:492] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:08.939975 1171 ts_tablet_manager.cc:1397] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250625 13:52:08.940887 1171 raft_consensus.cc:357] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.941088 1171 raft_consensus.cc:383] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.941146 1171 raft_consensus.cc:738] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.941350 1171 consensus_queue.cc:260] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:08.948064 887 tablet_service.cc:1468] Processing CreateTablet for tablet 93a0b94879874446b569b3cb3250b185 (DEFAULT_TABLE table=default.Table_53926d5babb045a4854877f904b639ec [id=325dd21568534ee58900ef0b8d79fae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.948935 888 tablet_service.cc:1468] Processing CreateTablet for tablet 53dda116b9ee425e9ce6181e217d53c4 (DEFAULT_TABLE table=default.Table_64adeba774054ea5a73be69a92bf079e [id=70185296dee54d82b614cd879436adbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.948930 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 53dda116b9ee425e9ce6181e217d53c4 (DEFAULT_TABLE table=default.Table_64adeba774054ea5a73be69a92bf079e [id=70185296dee54d82b614cd879436adbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.949553 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53dda116b9ee425e9ce6181e217d53c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.953049 2018 raft_consensus.cc:695] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:08.953397 2018 consensus_queue.cc:237] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:08.954258 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc0f3e2955344929b261f6affa378084 consensus state has no leader
I20250625 13:52:08.954689 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e3936edb9d8e47d08189e8903287c572"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.955255 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3936edb9d8e47d08189e8903287c572 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.955503 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a0b94879874446b569b3cb3250b185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.956327 591 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_f26987562c25452388d457eb3b600595 [id=62553d70f8ee470f944206ddc8cc2089] alter complete (version 1)
I20250625 13:52:08.956704 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_717dce88b94246cab98733e4565a27b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:08.957124 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_717dce88b94246cab98733e4565a27b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:08.957186 1166 tablet_bootstrap.cc:492] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.959458 1166 tablet_bootstrap.cc:654] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.965543 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c9a82adc5aa045f49e092dababafe4f8
I20250625 13:52:08.966063 590 catalog_manager.cc:6652] ReplaceTablet: tablet c9a82adc5aa045f49e092dababafe4f8 deleted and replaced by tablet 3562e8b8731a4d76bb1a36189f957e07
I20250625 13:52:08.966856 881 tablet_service.cc:1468] Processing CreateTablet for tablet c27caba6c4d14d51b10bf04f131c6a3c (DEFAULT_TABLE table=default.Table_8190e518ba344ab9b9081933923dae13 [id=1e19c242b306485dbce55fdc99fe5720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.967249 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27caba6c4d14d51b10bf04f131c6a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.968873 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cc3ae4d4f62d47af9a510dab1dd32ff5" } new_table_name: "default.Table_07265699458b42e8976c62a23fd2a135" modify_external_catalogs: true
I20250625 13:52:08.969079 744 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a82adc5aa045f49e092dababafe4f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3562e8b8731a4d76bb1a36189f957e07) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:08.969128 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53dda116b9ee425e9ce6181e217d53c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.969617 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 93a0b94879874446b569b3cb3250b185 (DEFAULT_TABLE table=default.Table_53926d5babb045a4854877f904b639ec [id=325dd21568534ee58900ef0b8d79fae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.970012 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93a0b94879874446b569b3cb3250b185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.970463 1710 raft_consensus.cc:2270] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:08.971025 1171 ts_tablet_manager.cc:1428] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20250625 13:52:08.972465 1171 tablet_bootstrap.cc:492] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:08.972652 1710 ts_tablet_manager.cc:1905] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:08.974872 1171 tablet_bootstrap.cc:654] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.981927 1166 tablet_bootstrap.cc:492] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:08.983006 1166 ts_tablet_manager.cc:1397] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.003s
I20250625 13:52:08.983729 1166 raft_consensus.cc:357] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.985538 1166 raft_consensus.cc:383] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:08.985678 1166 raft_consensus.cc:738] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:08.985999 1166 consensus_queue.cc:260] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:08.987251 1166 ts_tablet_manager.cc:1428] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:08.987635 1166 tablet_bootstrap.cc:492] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:08.988050 1168 ts_tablet_manager.cc:1918] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:08.988926 1168 log.cc:1199] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/637847cac1724bfeb629c4d4013396ca
I20250625 13:52:08.989009 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 9aadc67058554e87bb58d14b911333c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.989238 1168 ts_tablet_manager.cc:1939] T 637847cac1724bfeb629c4d4013396ca P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:08.989820 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet c666ff2d6b9b4ed4a18e370826c143fc (table default.Table_badc8f5a2d5040af96b9e7c9758c6417 [id=8a37f6c60b414741a615d1fa7f9a4f3c]) successfully deleted
I20250625 13:52:08.989919 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 694) took 3188 ms. Trace:
I20250625 13:52:08.990005 1168 rpcz_store.cc:276] 0625 13:52:05.801382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.801422 (+ 40us) service_pool.cc:225] Handling call
0625 13:52:08.898441 (+3097019us) ts_tablet_manager.cc:1145] Deleting tablet 637847cac1724bfeb629c4d4013396ca
0625 13:52:08.898449 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.944300 (+ 45851us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.987595 (+ 43295us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:08.989901 (+ 2306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3096870}
I20250625 13:52:08.990734 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 58aa3c21191e4bea9c5395d6e32b8717 (table default.Table_759e39cb7c154d6281c46789f97f0691 [id=5efdc94a9cf74a3daa545d4d93684eaa]) successfully deleted
I20250625 13:52:08.991714 881 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a82adc5aa045f49e092dababafe4f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3562e8b8731a4d76bb1a36189f957e07) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:08.992182 889 tablet_service.cc:1468] Processing CreateTablet for tablet 45079deff84a41b488ccdde2f5059431 (DEFAULT_TABLE table=default.Table_9b61484dc29b44ac9d0afd92e1da0ef3 [id=6cc418bddb944dc2b6d6a63452b6bfc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:08.993377 1166 tablet_bootstrap.cc:654] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:08.993927 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45079deff84a41b488ccdde2f5059431. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.995056 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 697) took 3179 ms. Trace:
I20250625 13:52:08.995165 1168 rpcz_store.cc:276] 0625 13:52:05.815606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.815668 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:08.994923 (+3179255us) ts_tablet_manager.cc:1145] Deleting tablet 43941422259b4413bc71acfce8928e1e
0625 13:52:08.994932 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:08.995036 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3179096}
I20250625 13:52:09.001379 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 696) took 3185 ms. Trace:
I20250625 13:52:09.001477 1168 rpcz_store.cc:276] 0625 13:52:05.815468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.816610 (+ 1142us) service_pool.cc:225] Handling call
0625 13:52:08.996799 (+3180189us) ts_tablet_manager.cc:1145] Deleting tablet ae4b428ae7bc49fcbc37e9d7d238ec21
0625 13:52:08.998117 (+ 1318us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.001359 (+ 3242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180005}
I20250625 13:52:09.002733 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 695) took 3187 ms. Trace:
I20250625 13:52:09.002821 1168 rpcz_store.cc:276] 0625 13:52:05.815355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.816828 (+ 1473us) service_pool.cc:225] Handling call
0625 13:52:09.002623 (+3185795us) ts_tablet_manager.cc:1145] Deleting tablet d3a17198b9a04138b24022cf8506ab36
0625 13:52:09.002633 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.002717 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3185695}
I20250625 13:52:09.004117 1015 tablet_service.cc:1468] Processing CreateTablet for tablet c27caba6c4d14d51b10bf04f131c6a3c (DEFAULT_TABLE table=default.Table_8190e518ba344ab9b9081933923dae13 [id=1e19c242b306485dbce55fdc99fe5720]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.004532 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27caba6c4d14d51b10bf04f131c6a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:08.931190 2143 tablet.cc:1722] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:09.007441 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 9aadc67058554e87bb58d14b911333c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.007649 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a82adc5aa045f49e092dababafe4f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3562e8b8731a4d76bb1a36189f957e07) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:09.007723 1027 consensus_meta.cc:287] T 28f180cd0a3141cc8c0889d7a8ca1d72 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent flushing consensus metadata: real 0.603s user 0.025s sys 0.000s
I20250625 13:52:09.011672 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet daa16c7ca9c7480dbd5fbd8114a449f2
I20250625 13:52:09.012420 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5d22cae7274d4d64811664493a162782"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.012995 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d22cae7274d4d64811664493a162782 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.014209 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e1b4da58289342d991cdf278005b96a0
I20250625 13:52:09.014786 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 39da3ebbe55f4c4fab4f8af444657b9b from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:09.016110 746 tablet_service.cc:1515] Processing DeleteTablet for tablet daa16c7ca9c7480dbd5fbd8114a449f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.016294 744 tablet_service.cc:1515] Processing DeleteTablet for tablet e1b4da58289342d991cdf278005b96a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.017351 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 698) took 3176 ms. Trace:
I20250625 13:52:09.017488 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 0e77736191544e8da32ca9927f0a5804 (table default.Table_b9f454bddd8a463b863eb0ca65ebf3af [id=7467e5b0513d42ff8c886b6080a5b539]) successfully deleted
W20250625 13:52:09.017768 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1d9bd351536c4558b189cebf8b793343 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d9bd351536c4558b189cebf8b793343 already in progress: creating tablet
I20250625 13:52:09.017853 1794 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33015
I20250625 13:52:09.017815 2157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33015 every 8 connection(s)
W20250625 13:52:09.018002 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9f3212d0b8de440e82156323ecb40a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f3212d0b8de440e82156323ecb40a7d already in progress: creating tablet
W20250625 13:52:09.018220 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 024f6cb363594420b0d5480fa79d156b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 024f6cb363594420b0d5480fa79d156b already in progress: creating tablet
W20250625 13:52:09.018411 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 81a359c798024b2a99aa3b307cc1967f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81a359c798024b2a99aa3b307cc1967f already in progress: creating tablet
I20250625 13:52:09.019791 888 tablet_service.cc:1515] Processing DeleteTablet for tablet e1b4da58289342d991cdf278005b96a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.020048 892 tablet_service.cc:1515] Processing DeleteTablet for tablet daa16c7ca9c7480dbd5fbd8114a449f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.020344 1794 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-0/data/info.pb
I20250625 13:52:09.021123 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu as pid 1794
W20250625 13:52:09.027511 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 43941422259b4413bc71acfce8928e1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43941422259b4413bc71acfce8928e1e already in progress: creating tablet
I20250625 13:52:09.029109 1165 ts_tablet_manager.cc:1918] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.017443 1168 rpcz_store.cc:276] 0625 13:52:05.840595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.840672 (+ 77us) service_pool.cc:225] Handling call
0625 13:52:09.017195 (+3176523us) ts_tablet_manager.cc:1145] Deleting tablet b4be0bc23cde4f0f94bd678096e0c7d8
0625 13:52:09.017209 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.017325 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3176291}
I20250625 13:52:09.029731 1168 tablet_replica.cc:331] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.030019 1168 raft_consensus.cc:2241] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.030203 1168 raft_consensus.cc:2270] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.030846 1168 ts_tablet_manager.cc:1905] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:09.030082 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ae4b428ae7bc49fcbc37e9d7d238ec21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae4b428ae7bc49fcbc37e9d7d238ec21 already in progress: creating tablet
W20250625 13:52:09.031231 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d3a17198b9a04138b24022cf8506ab36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a17198b9a04138b24022cf8506ab36 already in progress: creating tablet
I20250625 13:52:09.035488 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_458d788e23274f5d9f4044fe32959484"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.035653 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet b4be0bc23cde4f0f94bd678096e0c7d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4be0bc23cde4f0f94bd678096e0c7d8 already in progress: creating tablet
W20250625 13:52:09.036073 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_458d788e23274f5d9f4044fe32959484 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.037387 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet daa16c7ca9c7480dbd5fbd8114a449f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.037751 1165 log.cc:1199] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/430c3bed360b4f32855a8adb99076608
I20250625 13:52:09.038132 1165 ts_tablet_manager.cc:1939] T 430c3bed360b4f32855a8adb99076608 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:09.040406 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 430c3bed360b4f32855a8adb99076608 (table default.Table_1099b7a2614540d3ac476e1bad30554c [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]) successfully deleted
I20250625 13:52:09.041309 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 8b0e49ca4dee42c59c0a2927a47aa320 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:09.041569 1165 tablet_replica.cc:331] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:09.041596 1171 tablet_bootstrap.cc:492] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.041733 1171 ts_tablet_manager.cc:1397] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.000s
I20250625 13:52:09.041751 1165 raft_consensus.cc:2241] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.041914 1165 raft_consensus.cc:2270] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.042459 1171 raft_consensus.cc:357] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.042623 1171 raft_consensus.cc:383] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.042670 1171 raft_consensus.cc:738] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.042904 1171 consensus_queue.cc:260] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:09.043547 851 consensus_peers.cc:487] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36b151e200e1430c99ac194bc55881c1. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:09.043735 2147 raft_consensus.cc:491] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.043944 1165 ts_tablet_manager.cc:1905] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.043927 2147 raft_consensus.cc:513] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.045135 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:09.045626 711 leader_election.cc:343] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.046514 1171 ts_tablet_manager.cc:1428] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:09.046801 1171 tablet_bootstrap.cc:492] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.048575 1171 tablet_bootstrap.cc:654] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.050216 1842 consensus_queue.cc:1035] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:09.050251 591 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:09.058259 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet e1b4da58289342d991cdf278005b96a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.058259 1710 ts_tablet_manager.cc:1918] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.058790 1710 log.cc:1199] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:09.059134 1710 ts_tablet_manager.cc:1939] T 4fa801f7d98e4794b6ca439638f29e69 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:09.059684 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 4fa801f7d98e4794b6ca439638f29e69 (table default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328 [id=1e5117fcc9bb41c3bb98021f499b766c]) successfully deleted
I20250625 13:52:09.061112 2147 leader_election.cc:290] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:09.063017 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.063723 711 leader_election.cc:343] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.063860 711 leader_election.cc:304] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:09.064281 2147 raft_consensus.cc:2747] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.065228 1166 tablet_bootstrap.cc:492] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
W20250625 13:52:09.071589 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34371795c8094e73ab4e5dbeedb7aa2f consensus state has no leader
W20250625 13:52:09.072180 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet dc1c9cea3d5a405a958c6c7585dbef39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc1c9cea3d5a405a958c6c7585dbef39 already in progress: creating tablet
W20250625 13:52:09.073076 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c7ad5e90188847f38894fc3c1f8f2693 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7ad5e90188847f38894fc3c1f8f2693 already in progress: creating tablet
I20250625 13:52:09.073616 1166 ts_tablet_manager.cc:1397] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.004s
W20250625 13:52:09.076125 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9aadc67058554e87bb58d14b911333c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aadc67058554e87bb58d14b911333c4 already in progress: creating tablet
I20250625 13:52:09.077617 1165 ts_tablet_manager.cc:1918] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:09.077715 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c9a82adc5aa045f49e092dababafe4f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a82adc5aa045f49e092dababafe4f8 already in progress: creating tablet
I20250625 13:52:09.077867 1165 log.cc:1199] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/2404e712e18146b68757bd77362c1358
I20250625 13:52:09.078284 1165 ts_tablet_manager.cc:1939] T 2404e712e18146b68757bd77362c1358 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:52:09.080245 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet daa16c7ca9c7480dbd5fbd8114a449f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daa16c7ca9c7480dbd5fbd8114a449f2 already in progress: creating tablet
I20250625 13:52:09.080456 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 2404e712e18146b68757bd77362c1358 (table default.Table_4dc8c488c3e44f3f882049d1c0f8feb2 [id=0532faedda28481f9742008aecb6bfd3]) successfully deleted
I20250625 13:52:09.080746 2115 raft_consensus.cc:491] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.080884 2115 raft_consensus.cc:513] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:09.081324 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e1b4da58289342d991cdf278005b96a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1b4da58289342d991cdf278005b96a0 already in progress: creating tablet
I20250625 13:52:09.081472 2115 leader_election.cc:290] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:09.074608 1166 raft_consensus.cc:357] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.082280 1166 raft_consensus.cc:383] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.082345 1166 raft_consensus.cc:738] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.082538 1166 consensus_queue.cc:260] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.083062 1166 ts_tablet_manager.cc:1428] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 13:52:09.083362 1166 tablet_bootstrap.cc:492] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:09.083259 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0e81d4c904248eba858462bcf31c5d9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.083818 849 leader_election.cc:343] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:09.089704 851 leader_election.cc:343] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a0e81d4c904248eba858462bcf31c5d9
I20250625 13:52:09.089830 851 leader_election.cc:304] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:09.090276 1837 raft_consensus.cc:2747] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.093706 1170 tablet_bootstrap.cc:492] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:09.094292 1171 tablet_bootstrap.cc:492] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.094427 1171 ts_tablet_manager.cc:1397] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20250625 13:52:09.095124 1165 tablet_replica.cc:331] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:09.095238 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_893c6cc9fca542d0ab7c03d3f08207cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:09.095398 1165 raft_consensus.cc:2241] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.095563 1165 raft_consensus.cc:2270] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.095453 1171 raft_consensus.cc:357] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.095667 1171 raft_consensus.cc:383] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.095728 1171 raft_consensus.cc:738] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.095968 1171 consensus_queue.cc:260] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.096154 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_47e85ba24d824e5eaf587f4dbc7c9d22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.097956 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9aadc67058554e87bb58d14b911333c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aadc67058554e87bb58d14b911333c4 already in progress: creating tablet
I20250625 13:52:09.098654 1166 tablet_bootstrap.cc:654] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:09.099407 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47e85ba24d824e5eaf587f4dbc7c9d22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:09.101457 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_893c6cc9fca542d0ab7c03d3f08207cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.110028 2162 sys_catalog.cc:263] Verifying existing consensus state
I20250625 13:52:09.111553 2115 raft_consensus.cc:491] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.111689 2115 raft_consensus.cc:513] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.112000 1171 ts_tablet_manager.cc:1428] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 13:52:09.112257 2115 leader_election.cc:290] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:09.112339 1171 tablet_bootstrap.cc:492] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:09.113422 851 leader_election.cc:343] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9c1deb52366c404cbdc08076d4bcb3d1
I20250625 13:52:09.114671 1171 tablet_bootstrap.cc:654] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.116394 1168 ts_tablet_manager.cc:1918] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.116519 1168 log.cc:1199] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/29be8c0350584de382985e66556bdcca
I20250625 13:52:09.116858 1170 ts_tablet_manager.cc:1397] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.417s user 0.016s sys 0.024s
I20250625 13:52:09.103953 1165 ts_tablet_manager.cc:1905] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.117832 1170 raft_consensus.cc:357] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.119629 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c1deb52366c404cbdc08076d4bcb3d1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.123250 849 leader_election.cc:343] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.123397 849 leader_election.cc:304] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:09.123867 2115 raft_consensus.cc:2747] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.124117 1170 raft_consensus.cc:383] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.125439 1170 raft_consensus.cc:738] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.126277 1170 consensus_queue.cc:260] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.127224 2147 raft_consensus.cc:491] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.127408 2147 raft_consensus.cc:513] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.127696 1170 ts_tablet_manager.cc:1428] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250625 13:52:09.127983 1170 tablet_bootstrap.cc:492] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:09.129277 849 consensus_peers.cc:487] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36b151e200e1430c99ac194bc55881c1. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:09.132535 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 1ca3f3760e6e41a29d004e6afe63ec98 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:09.134203 1170 tablet_bootstrap.cc:654] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.134842 2147 leader_election.cc:290] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.134963 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:09.135959 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f56a8a6cb964d3a979e2f1d944eeddc" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:09.136133 912 raft_consensus.cc:2466] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:09.137003 711 leader_election.cc:304] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:09.137285 2147 raft_consensus.cc:2802] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:09.137370 2147 raft_consensus.cc:491] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:09.137436 2147 raft_consensus.cc:3058] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:09.140980 1165 ts_tablet_manager.cc:1918] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.141180 1165 log.cc:1199] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/024f6cb363594420b0d5480fa79d156b
I20250625 13:52:09.142482 1168 ts_tablet_manager.cc:1939] T 29be8c0350584de382985e66556bdcca P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.143222 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 699) took 3295 ms. Trace:
I20250625 13:52:09.143335 1168 rpcz_store.cc:276] 0625 13:52:05.847597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.847653 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:09.029664 (+3182011us) ts_tablet_manager.cc:1145] Deleting tablet 29be8c0350584de382985e66556bdcca
0625 13:52:09.029679 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.034007 (+ 4328us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.116296 (+ 82289us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.143201 (+ 26905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181814}
I20250625 13:52:09.143972 2147 raft_consensus.cc:513] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.144344 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 700) took 3268 ms. Trace:
I20250625 13:52:09.145632 2115 consensus_queue.cc:1035] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:09.144599 1168 rpcz_store.cc:276] 0625 13:52:05.875660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.875716 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:09.144084 (+3268368us) ts_tablet_manager.cc:1145] Deleting tablet 43b791fd11914591bb812cc155d950dc
0625 13:52:09.144095 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.144239 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3268173}
I20250625 13:52:09.147688 1168 tablet_replica.cc:331] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.148099 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f56a8a6cb964d3a979e2f1d944eeddc" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:09.148267 912 raft_consensus.cc:3058] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:09.148281 2147 leader_election.cc:290] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.148452 1168 raft_consensus.cc:2241] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.143894 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 29be8c0350584de382985e66556bdcca (table default.Table_13c3269d65ff41bc87efff42b2dce296 [id=1c58b83050f1494a91b0845e8dfe04bf]) successfully deleted
W20250625 13:52:09.149642 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 43b791fd11914591bb812cc155d950dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b791fd11914591bb812cc155d950dc already in progress: creating tablet
I20250625 13:52:09.153357 1165 ts_tablet_manager.cc:1939] T 024f6cb363594420b0d5480fa79d156b P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:09.154072 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0d7502063c8f4f06bcb3f4bd4c9e8cd9" } modify_external_catalogs: true
I20250625 13:52:09.154289 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0d7502063c8f4f06bcb3f4bd4c9e8cd9" } modify_external_catalogs: true
I20250625 13:52:09.154397 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9c9bfac37ad24311921d77442fc74013" } new_table_name: "default.Table_477135ae07a343758db03b263574b92b" modify_external_catalogs: true
I20250625 13:52:09.149288 1168 raft_consensus.cc:2270] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.152078 912 raft_consensus.cc:2466] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:09.154027 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_338af88ef39143f0823658cf2b61c83b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:09.156284 1168 ts_tablet_manager.cc:1905] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.156229 711 leader_election.cc:304] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:09.157383 2147 raft_consensus.cc:2802] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:52:09.157672 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_338af88ef39143f0823658cf2b61c83b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.158936 2147 raft_consensus.cc:695] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:09.159404 2147 consensus_queue.cc:237] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.161525 1171 tablet_bootstrap.cc:492] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.161661 1171 ts_tablet_manager.cc:1397] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
W20250625 13:52:09.162151 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 consensus state has no leader
I20250625 13:52:09.162672 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5c1dbe2671ff41d9b8d115b4b7ea58d6" } new_table_name: "default.Table_ce8a35cfcd0340f0ae0f19fd9ddef6a9" modify_external_catalogs: true
I20250625 13:52:09.162951 1171 raft_consensus.cc:357] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.163121 1171 raft_consensus.cc:383] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.163184 1171 raft_consensus.cc:738] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.163463 1171 consensus_queue.cc:260] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.135515 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f56a8a6cb964d3a979e2f1d944eeddc" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:09.148602 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f56a8a6cb964d3a979e2f1d944eeddc" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:09.167623 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d8906d18878c49f59fe57ee84374beb9" } new_table_name: "default.Table_4abaf88f43b74ca7b2cdc17a89443840" modify_external_catalogs: true
W20250625 13:52:09.169831 711 leader_election.cc:343] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:09.170176 711 leader_election.cc:343] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.171139 1171 ts_tablet_manager.cc:1428] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 13:52:09.171911 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 024f6cb363594420b0d5480fa79d156b (table default.Table_81c1b426fda24349805e3e5cf9fdd561 [id=3eae0d10bcb247e0a318d6ac81e8c5da]) successfully deleted
W20250625 13:52:09.172228 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c9a82adc5aa045f49e092dababafe4f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a82adc5aa045f49e092dababafe4f8 already in progress: creating tablet
W20250625 13:52:09.172488 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet daa16c7ca9c7480dbd5fbd8114a449f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daa16c7ca9c7480dbd5fbd8114a449f2 already in progress: creating tablet
W20250625 13:52:09.173904 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7221e009bda64b769846af1eb861d4fc consensus state has no leader
I20250625 13:52:09.174392 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c4265b6c17334e1fa2297818e6d2558a" } new_table_name: "default.Table_ce4451ff102a4c22ba9539daffd65aa6" modify_external_catalogs: true
I20250625 13:52:09.174631 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap starting.
W20250625 13:52:09.175048 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e1b4da58289342d991cdf278005b96a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1b4da58289342d991cdf278005b96a0 already in progress: creating tablet
I20250625 13:52:09.176240 1171 tablet_bootstrap.cc:492] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.178161 1171 tablet_bootstrap.cc:654] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.179682 1842 raft_consensus.cc:491] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.179824 1842 raft_consensus.cc:513] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.180446 1842 leader_election.cc:290] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:09.181658 851 leader_election.cc:343] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 77c50e1ebb9042ef8f0a651ba7186b74
I20250625 13:52:09.181980 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c50e1ebb9042ef8f0a651ba7186b74" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.182565 849 leader_election.cc:343] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.182682 849 leader_election.cc:304] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:09.183063 1842 raft_consensus.cc:2747] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.196393 1168 ts_tablet_manager.cc:1918] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.196553 1168 log.cc:1199] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/af05fba6fe2042aab5dd47b30e4dfcf3
I20250625 13:52:09.196913 1168 ts_tablet_manager.cc:1939] T af05fba6fe2042aab5dd47b30e4dfcf3 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.197710 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 701) took 3309 ms. Trace:
I20250625 13:52:09.197865 1166 tablet_bootstrap.cc:492] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:09.197825 1168 rpcz_store.cc:276] 0625 13:52:05.888734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.888786 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:09.147551 (+3258765us) ts_tablet_manager.cc:1145] Deleting tablet af05fba6fe2042aab5dd47b30e4dfcf3
0625 13:52:09.147560 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.159623 (+ 12063us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.196223 (+ 36600us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.197670 (+ 1447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258587}
I20250625 13:52:09.198000 1166 ts_tablet_manager.cc:1397] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.115s user 0.006s sys 0.001s
I20250625 13:52:09.198415 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet af05fba6fe2042aab5dd47b30e4dfcf3 (table default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]) successfully deleted
I20250625 13:52:09.198738 1166 raft_consensus.cc:357] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.198911 1166 raft_consensus.cc:383] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.198974 1166 raft_consensus.cc:738] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.199173 1166 consensus_queue.cc:260] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.207214 1166 ts_tablet_manager.cc:1428] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 13:52:09.207549 1166 tablet_bootstrap.cc:492] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:09.210361 1166 tablet_bootstrap.cc:654] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.214640 1168 tablet_replica.cc:331] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.214895 1168 raft_consensus.cc:2241] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.215119 1168 raft_consensus.cc:2270] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.215791 1168 ts_tablet_manager.cc:1905] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:09.233254 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5592ab667b04932896addf7514af24e consensus state has no leader
I20250625 13:52:09.233695 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d47cfb3324a443628070fd4437024a71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.234428 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d47cfb3324a443628070fd4437024a71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.236951 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:09.238827 1166 tablet_bootstrap.cc:492] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:09.239022 1166 ts_tablet_manager.cc:1397] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20250625 13:52:09.239920 1166 raft_consensus.cc:357] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.240157 1166 raft_consensus.cc:383] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.240284 1166 raft_consensus.cc:738] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.240401 1168 ts_tablet_manager.cc:1918] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.240582 1168 log.cc:1199] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/c0a3ea18bab944749d45da9b3139f8bc
I20250625 13:52:09.240636 1166 consensus_queue.cc:260] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.240948 1168 ts_tablet_manager.cc:1939] T c0a3ea18bab944749d45da9b3139f8bc P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.241485 1166 ts_tablet_manager.cc:1428] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:09.241680 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 702) took 3334 ms. Trace:
I20250625 13:52:09.241822 1166 tablet_bootstrap.cc:492] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:09.241787 1168 rpcz_store.cc:276] 0625 13:52:05.907470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:05.907544 (+ 74us) service_pool.cc:225] Handling call
0625 13:52:09.214563 (+3307019us) ts_tablet_manager.cc:1145] Deleting tablet c0a3ea18bab944749d45da9b3139f8bc
0625 13:52:09.214575 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.234255 (+ 19680us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.240284 (+ 6029us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.241618 (+ 1334us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3306808}
I20250625 13:52:09.242259 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet c0a3ea18bab944749d45da9b3139f8bc (table default.Table_c8b06a7351fb437aa0440f6fa3898939 [id=eac16bb1f501401c9f6803136dc04116]) successfully deleted
I20250625 13:52:09.243609 1166 tablet_bootstrap.cc:654] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.245890 1821 raft_consensus.cc:491] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.246065 1821 raft_consensus.cc:513] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.246845 1821 leader_election.cc:290] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.247841 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad4ea8cf44944b08f53428c95809daf" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:09.248071 912 raft_consensus.cc:2466] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:09.248008 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad4ea8cf44944b08f53428c95809daf" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:09.248211 777 raft_consensus.cc:2466] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:09.248553 983 leader_election.cc:304] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:09.249044 1821 raft_consensus.cc:2802] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:09.249130 1821 raft_consensus.cc:491] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:09.249199 1821 raft_consensus.cc:3058] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:09.253265 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 705) took 3233 ms. Trace:
I20250625 13:52:09.253377 1168 rpcz_store.cc:276] 0625 13:52:06.019917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.019971 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:09.253055 (+3233084us) ts_tablet_manager.cc:1145] Deleting tablet 5f49c0fc97d44b28acfd6cb493906edb
0625 13:52:09.253066 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.253181 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3232903}
I20250625 13:52:09.253839 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 708) took 3183 ms. Trace:
I20250625 13:52:09.253929 1168 rpcz_store.cc:276] 0625 13:52:06.070779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.070828 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:09.253671 (+3182843us) ts_tablet_manager.cc:1145] Deleting tablet d7c4a6a4fd574012818f3aa0a8b4d71f
0625 13:52:09.253699 (+ 28us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.253777 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3182675}
I20250625 13:52:09.254245 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 709) took 3182 ms. Trace:
I20250625 13:52:09.254333 1168 rpcz_store.cc:276] 0625 13:52:06.072222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.072273 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:09.254124 (+3181851us) ts_tablet_manager.cc:1145] Deleting tablet 17efe97d559a45dc9c0ca0b74efa7aa6
0625 13:52:09.254132 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.254198 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181699}
I20250625 13:52:09.254647 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 710) took 3165 ms. Trace:
I20250625 13:52:09.254732 1168 rpcz_store.cc:276] 0625 13:52:06.088726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.088780 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:09.254526 (+3165746us) ts_tablet_manager.cc:1145] Deleting tablet 614181c6e4d24e7b9e5da3b4818cf065
0625 13:52:09.254534 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.254598 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3165563}
I20250625 13:52:09.255105 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 713) took 3136 ms. Trace:
I20250625 13:52:09.255120 1821 raft_consensus.cc:513] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.255199 1168 rpcz_store.cc:276] 0625 13:52:06.118264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.118606 (+ 342us) service_pool.cc:225] Handling call
0625 13:52:09.254964 (+3136358us) ts_tablet_manager.cc:1145] Deleting tablet 6e147e09b5244a8b8f52e8d10c3db4c1
0625 13:52:09.254973 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.255045 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3136176}
I20250625 13:52:09.255517 1168 tablet_replica.cc:331] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.256343 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad4ea8cf44944b08f53428c95809daf" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
W20250625 13:52:09.256371 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 5f49c0fc97d44b28acfd6cb493906edb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f49c0fc97d44b28acfd6cb493906edb already in progress: creating tablet
I20250625 13:52:09.256528 777 raft_consensus.cc:3058] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:09.256649 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d7c4a6a4fd574012818f3aa0a8b4d71f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c4a6a4fd574012818f3aa0a8b4d71f already in progress: creating tablet
I20250625 13:52:09.256680 1821 leader_election.cc:290] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.256789 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad4ea8cf44944b08f53428c95809daf" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
W20250625 13:52:09.256872 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 17efe97d559a45dc9c0ca0b74efa7aa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17efe97d559a45dc9c0ca0b74efa7aa6 already in progress: creating tablet
I20250625 13:52:09.256955 912 raft_consensus.cc:3058] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:09.257050 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 614181c6e4d24e7b9e5da3b4818cf065 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 614181c6e4d24e7b9e5da3b4818cf065 already in progress: creating tablet
W20250625 13:52:09.257252 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6e147e09b5244a8b8f52e8d10c3db4c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e147e09b5244a8b8f52e8d10c3db4c1 already in progress: creating tablet
I20250625 13:52:09.257416 1168 raft_consensus.cc:2241] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.257625 1168 raft_consensus.cc:2270] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.258193 1168 ts_tablet_manager.cc:1905] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.259207 777 raft_consensus.cc:2466] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:09.259634 912 raft_consensus.cc:2466] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:09.259801 985 leader_election.cc:304] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:52:09.260177 1821 raft_consensus.cc:2802] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:09.261462 1171 tablet_bootstrap.cc:492] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.261672 1171 ts_tablet_manager.cc:1397] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.007s
I20250625 13:52:09.262476 1171 raft_consensus.cc:357] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.262670 1171 raft_consensus.cc:383] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.262742 1171 raft_consensus.cc:738] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.262970 1171 consensus_queue.cc:260] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.263520 1171 ts_tablet_manager.cc:1428] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:09.263919 1171 tablet_bootstrap.cc:492] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.271370 1821 raft_consensus.cc:695] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:09.271790 1821 consensus_queue.cc:237] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.276275 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
W20250625 13:52:09.279160 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6710249ce844de2a9e8b63ae65de9de has no consensus state
I20250625 13:52:09.279666 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b45de936fca240bd8d8d63cac7007f5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.280309 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b45de936fca240bd8d8d63cac7007f5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.282812 1171 tablet_bootstrap.cc:654] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.287930 1170 tablet_bootstrap.cc:492] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:09.288223 1170 ts_tablet_manager.cc:1397] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.002s
I20250625 13:52:09.288314 590 catalog_manager.cc:5582] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:09.288786 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
I20250625 13:52:09.289050 590 catalog_manager.cc:5582] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:09.289201 1170 raft_consensus.cc:357] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.289378 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet dc75a0a8421b457089cc707d9eb6b76f (table default.Table_f618b45e019044ea946538ed3fa05471 [id=fa0246aff3f648538e6fddcb30f3a399]). Expected version 3 got 0
I20250625 13:52:09.289505 1170 raft_consensus.cc:383] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.289608 1170 raft_consensus.cc:738] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.289659 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
I20250625 13:52:09.289848 590 catalog_manager.cc:5582] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:09.289958 1170 consensus_queue.cc:260] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.290750 1170 ts_tablet_manager.cc:1428] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:09.291653 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:09.292625 1170 tablet_bootstrap.cc:492] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:09.293869 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:09.294029 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:09.294346 1170 tablet_bootstrap.cc:654] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.294790 588 master_service.cc:948] ReplaceTablet: received request to replace tablet be3313ec5dfb41b68662d3ff4c8c94f9 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:09.296633 744 tablet_service.cc:1468] Processing CreateTablet for tablet 3670d1d409714821b06b98934d51762b (DEFAULT_TABLE table=default.Table_1e889135ee254afca2317cec6787455e [id=e2d387a341f140bdb4b9a113d0d84e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.297010 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3670d1d409714821b06b98934d51762b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.299082 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3670d1d409714821b06b98934d51762b (DEFAULT_TABLE table=default.Table_1e889135ee254afca2317cec6787455e [id=e2d387a341f140bdb4b9a113d0d84e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.299458 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3670d1d409714821b06b98934d51762b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.299607 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3670d1d409714821b06b98934d51762b (DEFAULT_TABLE table=default.Table_1e889135ee254afca2317cec6787455e [id=e2d387a341f140bdb4b9a113d0d84e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.299966 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3670d1d409714821b06b98934d51762b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.304289 746 tablet_service.cc:1468] Processing CreateTablet for tablet aa20e3830a74439a9ed54d2852f2528e (DEFAULT_TABLE table=default.Table_af34e284a9af4e60bc69bcdbf5acc655 [id=7ff0caf5022d45608adf85bc33cbff93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.304679 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa20e3830a74439a9ed54d2852f2528e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.304770 1015 tablet_service.cc:1468] Processing CreateTablet for tablet aa20e3830a74439a9ed54d2852f2528e (DEFAULT_TABLE table=default.Table_af34e284a9af4e60bc69bcdbf5acc655 [id=7ff0caf5022d45608adf85bc33cbff93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.305109 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa20e3830a74439a9ed54d2852f2528e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.305526 849 consensus_peers.cc:487] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1effed0fa10d409abcb31dbba54861d6. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:09.310609 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d1d81e81b06249259b90e4a9ea77cc3b (DEFAULT_TABLE table=default.Table_454bbba765254b009050c81e4147cb8e [id=6c9a2a8e6ba54fe7a3f95b5255e1a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.310963 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d81e81b06249259b90e4a9ea77cc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.312445 746 tablet_service.cc:1468] Processing CreateTablet for tablet d1d81e81b06249259b90e4a9ea77cc3b (DEFAULT_TABLE table=default.Table_454bbba765254b009050c81e4147cb8e [id=6c9a2a8e6ba54fe7a3f95b5255e1a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.312785 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d81e81b06249259b90e4a9ea77cc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.313582 751 tablet_service.cc:1468] Processing CreateTablet for tablet 784723a0249141f091c4cb6c1253d3f1 (DEFAULT_TABLE table=default.Table_30f7d431b5314e83b313365e3b0bffd5 [id=2f79cff5bb3e4be99f16ae477d3c6685]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.313910 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784723a0249141f091c4cb6c1253d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.314603 1168 ts_tablet_manager.cc:1918] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.314714 1168 log.cc:1199] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/fd39216b01054ff38ec9175d7a2f87f2
I20250625 13:52:09.315054 1168 ts_tablet_manager.cc:1939] T fd39216b01054ff38ec9175d7a2f87f2 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.316180 757 tablet_service.cc:1468] Processing CreateTablet for tablet 4acfcd35d6434ee59b9b0d831a59938c (DEFAULT_TABLE table=default.Table_6fc366fe7a4f4086a2f2cf60731c8089 [id=4333a35fefbe43199e8be03a9995aeb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.316356 887 tablet_service.cc:1468] Processing CreateTablet for tablet d1d81e81b06249259b90e4a9ea77cc3b (DEFAULT_TABLE table=default.Table_454bbba765254b009050c81e4147cb8e [id=6c9a2a8e6ba54fe7a3f95b5255e1a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.316459 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4acfcd35d6434ee59b9b0d831a59938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.316710 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1d81e81b06249259b90e4a9ea77cc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.317891 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 724) took 2846 ms. Trace:
I20250625 13:52:09.318001 1168 rpcz_store.cc:276] 0625 13:52:06.471461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.477208 (+ 5747us) service_pool.cc:225] Handling call
0625 13:52:09.255434 (+2778226us) ts_tablet_manager.cc:1145] Deleting tablet fd39216b01054ff38ec9175d7a2f87f2
0625 13:52:09.255442 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.295771 (+ 40329us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.314479 (+ 18708us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.317851 (+ 3372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2777907}
I20250625 13:52:09.318928 1168 tablet_replica.cc:331] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.319128 1168 raft_consensus.cc:2241] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.319281 1168 raft_consensus.cc:2270] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.319612 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1f44d0f54bc24feb8386cfce1cd67db6 (DEFAULT_TABLE table=default.Table_65d439aaf9c9480f8d1b9a3889e23a7e [id=899866b380154d069c3d01185eadda3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.319849 1168 ts_tablet_manager.cc:1905] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.319938 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f44d0f54bc24feb8386cfce1cd67db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.320734 1171 tablet_bootstrap.cc:492] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.320883 1171 ts_tablet_manager.cc:1397] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.003s
I20250625 13:52:09.321635 1171 raft_consensus.cc:357] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.321803 1171 raft_consensus.cc:383] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.321895 1171 raft_consensus.cc:738] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.322106 1171 consensus_queue.cc:260] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.322880 888 tablet_service.cc:1468] Processing CreateTablet for tablet 784723a0249141f091c4cb6c1253d3f1 (DEFAULT_TABLE table=default.Table_30f7d431b5314e83b313365e3b0bffd5 [id=2f79cff5bb3e4be99f16ae477d3c6685]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.323290 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784723a0249141f091c4cb6c1253d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.324093 757 tablet_service.cc:1468] Processing CreateTablet for tablet 57ee2b5f766c4561a3e4ff436ed33dc4 (DEFAULT_TABLE table=default.Table_3cb5cc5957974c9580239ecdf22f26cf [id=2e7bb5d0efcc4a17a42fe9c8435025ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.324576 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ee2b5f766c4561a3e4ff436ed33dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.327493 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 784723a0249141f091c4cb6c1253d3f1 (DEFAULT_TABLE table=default.Table_30f7d431b5314e83b313365e3b0bffd5 [id=2f79cff5bb3e4be99f16ae477d3c6685]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.327877 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 784723a0249141f091c4cb6c1253d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.328138 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet fd39216b01054ff38ec9175d7a2f87f2 (table default.Table_c54f67fabc114fc6bb555987a103adba [id=aa481ec0dc6e42b391641f4c3d05ca42]) successfully deleted
I20250625 13:52:09.328308 881 tablet_service.cc:1468] Processing CreateTablet for tablet 4acfcd35d6434ee59b9b0d831a59938c (DEFAULT_TABLE table=default.Table_6fc366fe7a4f4086a2f2cf60731c8089 [id=4333a35fefbe43199e8be03a9995aeb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.328693 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 57ee2b5f766c4561a3e4ff436ed33dc4 (DEFAULT_TABLE table=default.Table_3cb5cc5957974c9580239ecdf22f26cf [id=2e7bb5d0efcc4a17a42fe9c8435025ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.329007 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ee2b5f766c4561a3e4ff436ed33dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.330420 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4acfcd35d6434ee59b9b0d831a59938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.331545 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 4acfcd35d6434ee59b9b0d831a59938c (DEFAULT_TABLE table=default.Table_6fc366fe7a4f4086a2f2cf60731c8089 [id=4333a35fefbe43199e8be03a9995aeb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.331847 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4acfcd35d6434ee59b9b0d831a59938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.334430 1171 ts_tablet_manager.cc:1428] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 13:52:09.339138 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0d109dd2ff844dc5a6ec1dd6f1f0760e (DEFAULT_TABLE table=default.Table_efed4c6940fa46909feec8a3c6f600d7 [id=6a8281e96bd041899da767ee7293e5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.339423 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a8c4f90b40c344168e8f689c43101a64 (DEFAULT_TABLE table=default.Table_d2d334c69d834f9d89194a4ce4dc2fdd [id=8fc7fe0c2cbf43cfbef2138afa2e23bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.339589 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d109dd2ff844dc5a6ec1dd6f1f0760e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.339774 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c4f90b40c344168e8f689c43101a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.340293 741 tablet_service.cc:1468] Processing CreateTablet for tablet 38c954e01db443fc802d76bbbb853c14 (DEFAULT_TABLE table=default.Table_d0f8fc1edba14555a44cee33f16218ee [id=0226c50f914145e78e410c7cc4286589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.340716 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c954e01db443fc802d76bbbb853c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.342730 1166 tablet_bootstrap.cc:492] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:09.342862 1166 ts_tablet_manager.cc:1397] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.101s user 0.013s sys 0.000s
I20250625 13:52:09.343544 743 tablet_service.cc:1468] Processing CreateTablet for tablet ef1f9c03b0ef455cad05b3fe59bdd67c (DEFAULT_TABLE table=default.Table_b88679cb7cc74587927204b82e5915f4 [id=8a73adb9e4774a7fb5a53cd8896e2d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.343587 1166 raft_consensus.cc:357] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.343721 1166 raft_consensus.cc:383] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.343780 1166 raft_consensus.cc:738] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.343873 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1f9c03b0ef455cad05b3fe59bdd67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.343995 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 38c954e01db443fc802d76bbbb853c14 (DEFAULT_TABLE table=default.Table_d0f8fc1edba14555a44cee33f16218ee [id=0226c50f914145e78e410c7cc4286589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.344128 746 tablet_service.cc:1468] Processing CreateTablet for tablet 0d109dd2ff844dc5a6ec1dd6f1f0760e (DEFAULT_TABLE table=default.Table_efed4c6940fa46909feec8a3c6f600d7 [id=6a8281e96bd041899da767ee7293e5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.344287 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c954e01db443fc802d76bbbb853c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.344408 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d109dd2ff844dc5a6ec1dd6f1f0760e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.346136 1171 tablet_bootstrap.cc:492] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.347793 1171 tablet_bootstrap.cc:654] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.348372 752 tablet_service.cc:1468] Processing CreateTablet for tablet b312798103454d49a6c866062dfdbb15 (DEFAULT_TABLE table=default.Table_07d09c7961df461ea4c3ee75152fd7a2 [id=dbd86e9a581046659712a425187747c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.348693 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b312798103454d49a6c866062dfdbb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.348944 893 tablet_service.cc:1468] Processing CreateTablet for tablet ef1f9c03b0ef455cad05b3fe59bdd67c (DEFAULT_TABLE table=default.Table_b88679cb7cc74587927204b82e5915f4 [id=8a73adb9e4774a7fb5a53cd8896e2d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.349396 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f002689f9655413b9e1954d17fcfe92f (DEFAULT_TABLE table=default.Table_71df0e98edd34e26ab99f41178b13683 [id=cb470b1c27bb4880b240f9392faba82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.349756 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f002689f9655413b9e1954d17fcfe92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.351531 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ef1f9c03b0ef455cad05b3fe59bdd67c (DEFAULT_TABLE table=default.Table_b88679cb7cc74587927204b82e5915f4 [id=8a73adb9e4774a7fb5a53cd8896e2d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.351821 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1f9c03b0ef455cad05b3fe59bdd67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.353749 1166 consensus_queue.cc:260] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.354096 1020 tablet_service.cc:1468] Processing CreateTablet for tablet bfaedc53db264df181d0a56c0ed99c9d (DEFAULT_TABLE table=default.Table_98287f576a2f4435a569d69b7abf05f6 [id=fbe86be9782244dea61930c0a4ec16b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.354380 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaedc53db264df181d0a56c0ed99c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.355568 756 tablet_service.cc:1468] Processing CreateTablet for tablet f002689f9655413b9e1954d17fcfe92f (DEFAULT_TABLE table=default.Table_71df0e98edd34e26ab99f41178b13683 [id=cb470b1c27bb4880b240f9392faba82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.355886 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f002689f9655413b9e1954d17fcfe92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.357491 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 0d109dd2ff844dc5a6ec1dd6f1f0760e (DEFAULT_TABLE table=default.Table_efed4c6940fa46909feec8a3c6f600d7 [id=6a8281e96bd041899da767ee7293e5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.357669 884 tablet_service.cc:1468] Processing CreateTablet for tablet b312798103454d49a6c866062dfdbb15 (DEFAULT_TABLE table=default.Table_07d09c7961df461ea4c3ee75152fd7a2 [id=dbd86e9a581046659712a425187747c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.357827 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d109dd2ff844dc5a6ec1dd6f1f0760e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.359556 739 tablet_service.cc:1468] Processing CreateTablet for tablet bfaedc53db264df181d0a56c0ed99c9d (DEFAULT_TABLE table=default.Table_98287f576a2f4435a569d69b7abf05f6 [id=fbe86be9782244dea61930c0a4ec16b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.359966 1166 ts_tablet_manager.cc:1428] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 13:52:09.359937 1013 tablet_service.cc:1468] Processing CreateTablet for tablet b312798103454d49a6c866062dfdbb15 (DEFAULT_TABLE table=default.Table_07d09c7961df461ea4c3ee75152fd7a2 [id=dbd86e9a581046659712a425187747c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.360281 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b312798103454d49a6c866062dfdbb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.363648 891 tablet_service.cc:1468] Processing CreateTablet for tablet f002689f9655413b9e1954d17fcfe92f (DEFAULT_TABLE table=default.Table_71df0e98edd34e26ab99f41178b13683 [id=cb470b1c27bb4880b240f9392faba82e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.364215 896 tablet_service.cc:1468] Processing CreateTablet for tablet 9db10d184d7848b2a3990ab2f7b3939c (DEFAULT_TABLE table=default.Table_509942f9ab5846c88eabce5d076d1c35 [id=857b0bd2fcb64feb906166bbd2ea4ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.364234 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 9db10d184d7848b2a3990ab2f7b3939c (DEFAULT_TABLE table=default.Table_509942f9ab5846c88eabce5d076d1c35 [id=857b0bd2fcb64feb906166bbd2ea4ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.364583 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db10d184d7848b2a3990ab2f7b3939c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.364516 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 2d4ad6e59c2e48399b25be6d87abb1d2 (DEFAULT_TABLE table=default.Table_b14d3c37ddaa46468b60d59d70b9ccba [id=6dabbc3ed64c49c8a1669fa826cf0de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.364845 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4ad6e59c2e48399b25be6d87abb1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.364780 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2d4ad6e59c2e48399b25be6d87abb1d2 (DEFAULT_TABLE table=default.Table_b14d3c37ddaa46468b60d59d70b9ccba [id=6dabbc3ed64c49c8a1669fa826cf0de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.365139 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaedc53db264df181d0a56c0ed99c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.366142 886 tablet_service.cc:1468] Processing CreateTablet for tablet 17cf1172bcc74cca930824f9cda95f3b (DEFAULT_TABLE table=default.Table_a5675aff497c4f018385318a042343f4 [id=d24e8f65d84743ada133d7facbb937db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.367560 888 tablet_service.cc:1468] Processing CreateTablet for tablet 38c954e01db443fc802d76bbbb853c14 (DEFAULT_TABLE table=default.Table_d0f8fc1edba14555a44cee33f16218ee [id=0226c50f914145e78e410c7cc4286589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.368235 889 tablet_service.cc:1468] Processing CreateTablet for tablet aa20e3830a74439a9ed54d2852f2528e (DEFAULT_TABLE table=default.Table_af34e284a9af4e60bc69bcdbf5acc655 [id=7ff0caf5022d45608adf85bc33cbff93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.368793 887 tablet_service.cc:1468] Processing CreateTablet for tablet 57ee2b5f766c4561a3e4ff436ed33dc4 (DEFAULT_TABLE table=default.Table_3cb5cc5957974c9580239ecdf22f26cf [id=2e7bb5d0efcc4a17a42fe9c8435025ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.375674 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1f44d0f54bc24feb8386cfce1cd67db6 (DEFAULT_TABLE table=default.Table_65d439aaf9c9480f8d1b9a3889e23a7e [id=899866b380154d069c3d01185eadda3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.376215 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 17cf1172bcc74cca930824f9cda95f3b (DEFAULT_TABLE table=default.Table_a5675aff497c4f018385318a042343f4 [id=d24e8f65d84743ada133d7facbb937db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.376658 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17cf1172bcc74cca930824f9cda95f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.377209 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 55a254d82f764aa2860e465b4857a413 (DEFAULT_TABLE table=default.Table_f7f814de767d41a4b070ac5045680405 [id=ea8051b008d449abaff6f7a9b8a71493]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.377774 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a254d82f764aa2860e465b4857a413. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.378454 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3c1061dac55a4e97ad139950a7c8f302" } modify_external_catalogs: true
I20250625 13:52:09.378679 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3c1061dac55a4e97ad139950a7c8f302" } modify_external_catalogs: true
I20250625 13:52:09.379980 894 tablet_service.cc:1468] Processing CreateTablet for tablet 55a254d82f764aa2860e465b4857a413 (DEFAULT_TABLE table=default.Table_f7f814de767d41a4b070ac5045680405 [id=ea8051b008d449abaff6f7a9b8a71493]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.380542 878 tablet_service.cc:1468] Processing CreateTablet for tablet e99b7145ea274ef2a8d4bef4fb4e16ad (DEFAULT_TABLE table=default.Table_eef69b729d1a490a8704b781d33e575e [id=f6b3315673b942cbbfd7a7ab3d9f71d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.380589 892 tablet_service.cc:1468] Processing CreateTablet for tablet a8c4f90b40c344168e8f689c43101a64 (DEFAULT_TABLE table=default.Table_d2d334c69d834f9d89194a4ce4dc2fdd [id=8fc7fe0c2cbf43cfbef2138afa2e23bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.382062 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e99b7145ea274ef2a8d4bef4fb4e16ad (DEFAULT_TABLE table=default.Table_eef69b729d1a490a8704b781d33e575e [id=f6b3315673b942cbbfd7a7ab3d9f71d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.382941 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99b7145ea274ef2a8d4bef4fb4e16ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.383636 740 tablet_service.cc:1468] Processing CreateTablet for tablet 9db10d184d7848b2a3990ab2f7b3939c (DEFAULT_TABLE table=default.Table_509942f9ab5846c88eabce5d076d1c35 [id=857b0bd2fcb64feb906166bbd2ea4ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.384232 745 tablet_service.cc:1468] Processing CreateTablet for tablet 2d4ad6e59c2e48399b25be6d87abb1d2 (DEFAULT_TABLE table=default.Table_b14d3c37ddaa46468b60d59d70b9ccba [id=6dabbc3ed64c49c8a1669fa826cf0de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.384773 747 tablet_service.cc:1468] Processing CreateTablet for tablet 17cf1172bcc74cca930824f9cda95f3b (DEFAULT_TABLE table=default.Table_a5675aff497c4f018385318a042343f4 [id=d24e8f65d84743ada133d7facbb937db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.384972 890 tablet_service.cc:1468] Processing CreateTablet for tablet 913bc2f716c14de890bf1efdb0611dde (DEFAULT_TABLE table=default.Table_c640e0bdbd3e4274ab75511cebd6828d [id=6442e9a56ec74a88a4cad1855a4039eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.385330 749 tablet_service.cc:1468] Processing CreateTablet for tablet 55a254d82f764aa2860e465b4857a413 (DEFAULT_TABLE table=default.Table_f7f814de767d41a4b070ac5045680405 [id=ea8051b008d449abaff6f7a9b8a71493]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.385860 750 tablet_service.cc:1468] Processing CreateTablet for tablet e99b7145ea274ef2a8d4bef4fb4e16ad (DEFAULT_TABLE table=default.Table_eef69b729d1a490a8704b781d33e575e [id=f6b3315673b942cbbfd7a7ab3d9f71d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.386428 742 tablet_service.cc:1468] Processing CreateTablet for tablet 913bc2f716c14de890bf1efdb0611dde (DEFAULT_TABLE table=default.Table_c640e0bdbd3e4274ab75511cebd6828d [id=6442e9a56ec74a88a4cad1855a4039eb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:09.386874 851 consensus_peers.cc:487] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1effed0fa10d409abcb31dbba54861d6. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:09.388152 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 913bc2f716c14de890bf1efdb0611dde (DEFAULT_TABLE table=default.Table_c640e0bdbd3e4274ab75511cebd6828d [id=6442e9a56ec74a88a4cad1855a4039eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.388327 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 1a455c3bdafb4e9bbb79ab4b2ae39399 (DEFAULT_TABLE table=default.Table_237174b0502449feae068850a65184a4 [id=69ad2735661f4e419c99c1f16b53ae69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.389178 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913bc2f716c14de890bf1efdb0611dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.390246 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a455c3bdafb4e9bbb79ab4b2ae39399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.393635 885 tablet_service.cc:1468] Processing CreateTablet for tablet 1a455c3bdafb4e9bbb79ab4b2ae39399 (DEFAULT_TABLE table=default.Table_237174b0502449feae068850a65184a4 [id=69ad2735661f4e419c99c1f16b53ae69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.403295 754 tablet_service.cc:1468] Processing CreateTablet for tablet 1a455c3bdafb4e9bbb79ab4b2ae39399 (DEFAULT_TABLE table=default.Table_237174b0502449feae068850a65184a4 [id=69ad2735661f4e419c99c1f16b53ae69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.404132 753 tablet_service.cc:1468] Processing CreateTablet for tablet 3c561fcc70924ea5ab4ab2b569df215a (DEFAULT_TABLE table=default.Table_ab888f08b1984d8da2a92b8b396d4989 [id=13b265cdb0214da085594e59880edfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.404788 879 tablet_service.cc:1468] Processing CreateTablet for tablet 3c561fcc70924ea5ab4ab2b569df215a (DEFAULT_TABLE table=default.Table_ab888f08b1984d8da2a92b8b396d4989 [id=13b265cdb0214da085594e59880edfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.406122 751 tablet_service.cc:1468] Processing CreateTablet for tablet a8c4f90b40c344168e8f689c43101a64 (DEFAULT_TABLE table=default.Table_d2d334c69d834f9d89194a4ce4dc2fdd [id=8fc7fe0c2cbf43cfbef2138afa2e23bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.407548 1166 tablet_bootstrap.cc:492] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:09.410372 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db10d184d7848b2a3990ab2f7b3939c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.411520 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4ad6e59c2e48399b25be6d87abb1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.411702 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a254d82f764aa2860e465b4857a413. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.415498 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99b7145ea274ef2a8d4bef4fb4e16ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.416267 1166 tablet_bootstrap.cc:654] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.417279 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913bc2f716c14de890bf1efdb0611dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.417465 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17cf1172bcc74cca930824f9cda95f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.418007 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f44d0f54bc24feb8386cfce1cd67db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.419641 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1f9c03b0ef455cad05b3fe59bdd67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.419878 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f002689f9655413b9e1954d17fcfe92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.420601 880 tablet_service.cc:1468] Processing CreateTablet for tablet bfaedc53db264df181d0a56c0ed99c9d (DEFAULT_TABLE table=default.Table_98287f576a2f4435a569d69b7abf05f6 [id=fbe86be9782244dea61930c0a4ec16b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.420921 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b312798103454d49a6c866062dfdbb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.421139 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 913bc2f716c14de890bf1efdb0611dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.421120 2147 raft_consensus.cc:491] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.421471 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c561fcc70924ea5ab4ab2b569df215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.425949 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a455c3bdafb4e9bbb79ab4b2ae39399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.426843 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c954e01db443fc802d76bbbb853c14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.427459 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ee2b5f766c4561a3e4ff436ed33dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.429267 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 1f44d0f54bc24feb8386cfce1cd67db6 (DEFAULT_TABLE table=default.Table_65d439aaf9c9480f8d1b9a3889e23a7e [id=899866b380154d069c3d01185eadda3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.429616 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f44d0f54bc24feb8386cfce1cd67db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.430814 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfaedc53db264df181d0a56c0ed99c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.431512 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17cf1172bcc74cca930824f9cda95f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.433045 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa20e3830a74439a9ed54d2852f2528e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.433818 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a455c3bdafb4e9bbb79ab4b2ae39399. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.434257 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c561fcc70924ea5ab4ab2b569df215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.436658 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a254d82f764aa2860e465b4857a413. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.433400 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 3c561fcc70924ea5ab4ab2b569df215a (DEFAULT_TABLE table=default.Table_ab888f08b1984d8da2a92b8b396d4989 [id=13b265cdb0214da085594e59880edfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.437098 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c561fcc70924ea5ab4ab2b569df215a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.421231 2147 raft_consensus.cc:513] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.440439 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4ad6e59c2e48399b25be6d87abb1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.440753 2147 leader_election.cc:290] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.441335 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c5fc63828d9408b9cf774a966f43cf2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:09.442436 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c5fc63828d9408b9cf774a966f43cf2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:09.442931 711 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.445992 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1b1feaf6e78b4dccbdb3ddc0369de74b (DEFAULT_TABLE table=default.Table_e96b4c55e0434858b439e40c14a99aca [id=93bd983dac6d41eba4f5854fa7049aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.446362 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1feaf6e78b4dccbdb3ddc0369de74b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.447090 711 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.447206 711 leader_election.cc:304] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:09.447666 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e99b7145ea274ef2a8d4bef4fb4e16ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.450932 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db10d184d7848b2a3990ab2f7b3939c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.453158 2147 raft_consensus.cc:2747] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.457643 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c4f90b40c344168e8f689c43101a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.462597 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c4f90b40c344168e8f689c43101a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.464273 886 tablet_service.cc:1468] Processing CreateTablet for tablet 1b1feaf6e78b4dccbdb3ddc0369de74b (DEFAULT_TABLE table=default.Table_e96b4c55e0434858b439e40c14a99aca [id=93bd983dac6d41eba4f5854fa7049aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.464748 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1feaf6e78b4dccbdb3ddc0369de74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.466554 742 tablet_service.cc:1468] Processing CreateTablet for tablet 1b1feaf6e78b4dccbdb3ddc0369de74b (DEFAULT_TABLE table=default.Table_e96b4c55e0434858b439e40c14a99aca [id=93bd983dac6d41eba4f5854fa7049aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.466940 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b1feaf6e78b4dccbdb3ddc0369de74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.466554 749 tablet_service.cc:1468] Processing CreateTablet for tablet 9d6441a11bb6425d9626450122230dd1 (DEFAULT_TABLE table=default.Table_e3977ec02e244d9c8c7b932dbf431c53 [id=cfa447f89dc2461ba562e4d064b981d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.467723 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6441a11bb6425d9626450122230dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.474026 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 9d6441a11bb6425d9626450122230dd1 (DEFAULT_TABLE table=default.Table_e3977ec02e244d9c8c7b932dbf431c53 [id=cfa447f89dc2461ba562e4d064b981d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.474718 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6441a11bb6425d9626450122230dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.481791 877 tablet_service.cc:1468] Processing CreateTablet for tablet 9d6441a11bb6425d9626450122230dd1 (DEFAULT_TABLE table=default.Table_e3977ec02e244d9c8c7b932dbf431c53 [id=cfa447f89dc2461ba562e4d064b981d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.482193 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6441a11bb6425d9626450122230dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.482779 1166 tablet_bootstrap.cc:492] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:09.483052 1166 ts_tablet_manager.cc:1397] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.000s
I20250625 13:52:09.483907 1166 raft_consensus.cc:357] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.484431 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 0b5ae05ecd414957a61d466a681d3016 (DEFAULT_TABLE table=default.Table_9bfb4aea9df24975a81b278ceadf7542 [id=a05f17b36110454585dd0ac9003d23e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.484956 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b5ae05ecd414957a61d466a681d3016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.486889 1166 raft_consensus.cc:383] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.487041 1166 raft_consensus.cc:738] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.487339 1166 consensus_queue.cc:260] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.487524 1842 raft_consensus.cc:491] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.487664 1842 raft_consensus.cc:513] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.488138 1166 ts_tablet_manager.cc:1428] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:52:09.488306 1842 leader_election.cc:290] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:09.488471 1166 tablet_bootstrap.cc:492] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:09.488785 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be3313ec5dfb41b68662d3ff4c8c94f9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.489292 849 leader_election.cc:343] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.489876 1166 tablet_bootstrap.cc:654] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:09.491171 851 leader_election.cc:343] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: be3313ec5dfb41b68662d3ff4c8c94f9
I20250625 13:52:09.491273 851 leader_election.cc:304] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:09.491919 1842 raft_consensus.cc:2747] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.501478 1171 tablet_bootstrap.cc:492] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.501628 1171 ts_tablet_manager.cc:1397] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.156s user 0.003s sys 0.000s
I20250625 13:52:09.502293 1171 raft_consensus.cc:357] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.502467 1171 raft_consensus.cc:383] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.502519 1171 raft_consensus.cc:738] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.502748 1171 consensus_queue.cc:260] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.506018 1171 ts_tablet_manager.cc:1428] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:09.506312 1171 tablet_bootstrap.cc:492] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.507730 1171 tablet_bootstrap.cc:654] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.508864 1166 tablet_bootstrap.cc:492] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:09.509004 1166 ts_tablet_manager.cc:1397] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20250625 13:52:09.509846 1166 raft_consensus.cc:357] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.510036 1166 raft_consensus.cc:383] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.510100 1166 raft_consensus.cc:738] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.510332 1166 consensus_queue.cc:260] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.510841 1166 ts_tablet_manager.cc:1428] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:09.511128 1166 tablet_bootstrap.cc:492] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:09.513939 1168 ts_tablet_manager.cc:1918] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.514078 1168 log.cc:1199] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/2cd5303e674246ada7d36574b535b333
I20250625 13:52:09.514396 1168 ts_tablet_manager.cc:1939] T 2cd5303e674246ada7d36574b535b333 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.515975 879 tablet_service.cc:1468] Processing CreateTablet for tablet 0b5ae05ecd414957a61d466a681d3016 (DEFAULT_TABLE table=default.Table_9bfb4aea9df24975a81b278ceadf7542 [id=a05f17b36110454585dd0ac9003d23e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.516460 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b5ae05ecd414957a61d466a681d3016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.527838 1170 tablet_bootstrap.cc:492] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:09.527998 1170 ts_tablet_manager.cc:1397] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.235s user 0.001s sys 0.002s
I20250625 13:52:09.528750 1170 raft_consensus.cc:357] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.528929 1170 raft_consensus.cc:383] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.528990 1170 raft_consensus.cc:738] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.529201 1170 consensus_queue.cc:260] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.531848 1170 ts_tablet_manager.cc:1428] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 13:52:09.532135 1170 tablet_bootstrap.cc:492] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:09.533684 1170 tablet_bootstrap.cc:654] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.535717 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8b0e49ca4dee42c59c0a2927a47aa320
I20250625 13:52:09.536247 594 catalog_manager.cc:6652] ReplaceTablet: tablet 8b0e49ca4dee42c59c0a2927a47aa320 deleted and replaced by tablet a1263945be894618a7f9946e7ca50efd
I20250625 13:52:09.536950 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:09.537335 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b0e49ca4dee42c59c0a2927a47aa320 with delete_type TABLET_DATA_DELETED (replaced by tablet a1263945be894618a7f9946e7ca50efd) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.538419 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:09.539147 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6dd84b8afa2548fe8ce5925f0bf806f0
I20250625 13:52:09.540275 1171 tablet_bootstrap.cc:492] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.540414 1171 ts_tablet_manager.cc:1397] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.002s
I20250625 13:52:09.540984 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b0e49ca4dee42c59c0a2927a47aa320 with delete_type TABLET_DATA_DELETED (replaced by tablet a1263945be894618a7f9946e7ca50efd) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.541354 1171 raft_consensus.cc:357] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.541536 1171 raft_consensus.cc:383] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.541594 1171 raft_consensus.cc:738] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.541821 1171 consensus_queue.cc:260] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.544288 1171 ts_tablet_manager.cc:1428] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 13:52:09.547416 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 725) took 3042 ms. Trace:
I20250625 13:52:09.547509 1168 rpcz_store.cc:276] 0625 13:52:06.504755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.513227 (+ 8472us) service_pool.cc:225] Handling call
0625 13:52:09.318867 (+2805640us) ts_tablet_manager.cc:1145] Deleting tablet 2cd5303e674246ada7d36574b535b333
0625 13:52:09.318874 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.464649 (+145775us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.512309 (+ 47660us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.547396 (+ 35087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2805387}
I20250625 13:52:09.548065 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 727) took 3004 ms. Trace:
I20250625 13:52:09.548575 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:09.548835 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8b0e49ca4dee42c59c0a2927a47aa320 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b0e49ca4dee42c59c0a2927a47aa320 already in progress: creating tablet
I20250625 13:52:09.549168 892 tablet_service.cc:1468] Processing CreateTablet for tablet 13e4738b2e204bb1bc53908429537d8b (DEFAULT_TABLE table=default.Table_a0aeb065271449cb97100a02d36b89db [id=d0b6108df9b747169ededc903b7f1841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.549629 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e4738b2e204bb1bc53908429537d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.549914 1014 tablet_service.cc:1468] Processing CreateTablet for tablet ea393399f6264eba8922c4fc1b1af53d (DEFAULT_TABLE table=default.Table_21e1c6c522e04d72bb8739a9b7ef211e [id=59bd6453753f4c5eaa67a671135309d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.550177 1011 tablet_service.cc:1468] Processing CreateTablet for tablet a44fab4e30564e9fbbc8d2b351437052 (DEFAULT_TABLE table=default.Table_87f21414bdc44bddaaa231cd2dfc7ecf [id=27bb5b5377e2489c9fda8d0d776ef41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.550416 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea393399f6264eba8922c4fc1b1af53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.550415 894 tablet_service.cc:1468] Processing CreateTablet for tablet a44fab4e30564e9fbbc8d2b351437052 (DEFAULT_TABLE table=default.Table_87f21414bdc44bddaaa231cd2dfc7ecf [id=27bb5b5377e2489c9fda8d0d776ef41e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:09.550601 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7221e009bda64b769846af1eb861d4fc (table default.Table_ce8a35cfcd0340f0ae0f19fd9ddef6a9 [id=e2779d18910847aa878f92c43845d557], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7221e009bda64b769846af1eb861d4fc consensus state has no leader
I20250625 13:52:09.550745 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a44fab4e30564e9fbbc8d2b351437052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.550724 1027 tablet_service.cc:1468] Processing CreateTablet for tablet f03370056798484192cf6ee19e124bc6 (DEFAULT_TABLE table=default.Table_3cf16642852d4afa96c4425698093888 [id=3a65e259b60046d094c1ab12973ecffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.551105 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03370056798484192cf6ee19e124bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.551182 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 2cd5303e674246ada7d36574b535b333 (table default.Table_09cd14059fee4f4aaadc9c4e4bd5a249 [id=a79073f0368145dcab2528663f56e9aa]) successfully deleted
I20250625 13:52:09.553906 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 13e4738b2e204bb1bc53908429537d8b (DEFAULT_TABLE table=default.Table_a0aeb065271449cb97100a02d36b89db [id=d0b6108df9b747169ededc903b7f1841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.554237 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e4738b2e204bb1bc53908429537d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.554459 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a44fab4e30564e9fbbc8d2b351437052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.554966 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.549958 881 tablet_service.cc:1468] Processing CreateTablet for tablet ea393399f6264eba8922c4fc1b1af53d (DEFAULT_TABLE table=default.Table_21e1c6c522e04d72bb8739a9b7ef211e [id=59bd6453753f4c5eaa67a671135309d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.555606 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea393399f6264eba8922c4fc1b1af53d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.555977 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dd84b8afa2548fe8ce5925f0bf806f0 already in progress: creating tablet
I20250625 13:52:09.557068 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b0e49ca4dee42c59c0a2927a47aa320 with delete_type TABLET_DATA_DELETED (replaced by tablet a1263945be894618a7f9946e7ca50efd) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.558125 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0b5ae05ecd414957a61d466a681d3016 (DEFAULT_TABLE table=default.Table_9bfb4aea9df24975a81b278ceadf7542 [id=a05f17b36110454585dd0ac9003d23e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.558535 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b5ae05ecd414957a61d466a681d3016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.559949 1166 tablet_bootstrap.cc:654] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.562186 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 024f6cb363594420b0d5480fa79d156b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.562749 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.562769 747 tablet_service.cc:1468] Processing CreateTablet for tablet f03370056798484192cf6ee19e124bc6 (DEFAULT_TABLE table=default.Table_3cf16642852d4afa96c4425698093888 [id=3a65e259b60046d094c1ab12973ecffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.563189 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03370056798484192cf6ee19e124bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.563143 741 tablet_service.cc:1468] Processing CreateTablet for tablet ea393399f6264eba8922c4fc1b1af53d (DEFAULT_TABLE table=default.Table_21e1c6c522e04d72bb8739a9b7ef211e [id=59bd6453753f4c5eaa67a671135309d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.563548 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea393399f6264eba8922c4fc1b1af53d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.563529 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8b0e49ca4dee42c59c0a2927a47aa320 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b0e49ca4dee42c59c0a2927a47aa320 already in progress: creating tablet
W20250625 13:52:09.563853 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 024f6cb363594420b0d5480fa79d156b because the tablet was not found. No further retry: Not found: Tablet not found: 024f6cb363594420b0d5480fa79d156b
I20250625 13:52:09.565699 753 tablet_service.cc:1468] Processing CreateTablet for tablet a44fab4e30564e9fbbc8d2b351437052 (DEFAULT_TABLE table=default.Table_87f21414bdc44bddaaa231cd2dfc7ecf [id=27bb5b5377e2489c9fda8d0d776ef41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.566054 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a44fab4e30564e9fbbc8d2b351437052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.566474 1165 tablet_replica.cc:331] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:09.566650 1165 raft_consensus.cc:2241] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:09.566900 1165 pending_rounds.cc:70] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:52:09.566953 1165 pending_rounds.cc:77] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520630596431872 op_type: NO_OP noop_request { }
I20250625 13:52:09.567045 1165 raft_consensus.cc:2887] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:09.567147 1165 raft_consensus.cc:2270] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.548142 1168 rpcz_store.cc:276] 0625 13:52:06.543442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.543510 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:09.547935 (+3004425us) ts_tablet_manager.cc:1145] Deleting tablet a028df315713443986c4b362ed17fd99
0625 13:52:09.547946 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.548047 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004235}
I20250625 13:52:09.569963 877 tablet_service.cc:1468] Processing CreateTablet for tablet f03370056798484192cf6ee19e124bc6 (DEFAULT_TABLE table=default.Table_3cf16642852d4afa96c4425698093888 [id=3a65e259b60046d094c1ab12973ecffd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:09.570202 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a028df315713443986c4b362ed17fd99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a028df315713443986c4b362ed17fd99 already in progress: creating tablet
I20250625 13:52:09.570326 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f03370056798484192cf6ee19e124bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.570914 751 tablet_service.cc:1468] Processing CreateTablet for tablet 13e4738b2e204bb1bc53908429537d8b (DEFAULT_TABLE table=default.Table_a0aeb065271449cb97100a02d36b89db [id=d0b6108df9b747169ededc903b7f1841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.571332 1165 ts_tablet_manager.cc:1905] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.571251 1011 tablet_service.cc:1468] Processing CreateTablet for tablet bf962c47820f45e79e0e70f9208a17ff (DEFAULT_TABLE table=default.Table_5bc830734d3046dcafc7d1ccc0c5c561 [id=7b3e78773eaa40e6b13090408bb266c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.572118 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf962c47820f45e79e0e70f9208a17ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.572813 896 tablet_service.cc:1468] Processing CreateTablet for tablet bf962c47820f45e79e0e70f9208a17ff (DEFAULT_TABLE table=default.Table_5bc830734d3046dcafc7d1ccc0c5c561 [id=7b3e78773eaa40e6b13090408bb266c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.573127 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf962c47820f45e79e0e70f9208a17ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.576867 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 728) took 3025 ms. Trace:
I20250625 13:52:09.576960 1168 rpcz_store.cc:276] 0625 13:52:06.551544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.552578 (+ 1034us) service_pool.cc:225] Handling call
0625 13:52:09.576742 (+3024164us) ts_tablet_manager.cc:1145] Deleting tablet b14d00c93a964edd815122e3723f89e7
0625 13:52:09.576752 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.576851 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3023973}
I20250625 13:52:09.577425 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 729) took 3008 ms. Trace:
I20250625 13:52:09.577498 1168 rpcz_store.cc:276] 0625 13:52:06.568892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.568955 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:09.577329 (+3008374us) ts_tablet_manager.cc:1145] Deleting tablet 8e2bc4df39674dc3be0cd32c68f1ebca
0625 13:52:09.577338 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.577410 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3008116}
I20250625 13:52:09.577906 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 730) took 2982 ms. Trace:
I20250625 13:52:09.577986 1168 rpcz_store.cc:276] 0625 13:52:06.595098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.595168 (+ 70us) service_pool.cc:225] Handling call
0625 13:52:09.577815 (+2982647us) ts_tablet_manager.cc:1145] Deleting tablet c431abffd05c4136b7510c60eee2a8b1
0625 13:52:09.577824 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.577891 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2982429}
I20250625 13:52:09.578394 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 731) took 2946 ms. Trace:
I20250625 13:52:09.578465 1168 rpcz_store.cc:276] 0625 13:52:06.632156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.633202 (+ 1046us) service_pool.cc:225] Handling call
0625 13:52:09.578308 (+2945106us) ts_tablet_manager.cc:1145] Deleting tablet 68a7712455694a1cbead413230620827
0625 13:52:09.578317 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.578381 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2944923}
I20250625 13:52:09.579171 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 732) took 2946 ms. Trace:
I20250625 13:52:09.579243 1168 rpcz_store.cc:276] 0625 13:52:06.632243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.633421 (+ 1178us) service_pool.cc:225] Handling call
0625 13:52:09.579076 (+2945655us) ts_tablet_manager.cc:1145] Deleting tablet c366da4fb138453386d97fba0f5c91be
0625 13:52:09.579084 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.579157 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2945499}
I20250625 13:52:09.579588 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 733) took 2947 ms. Trace:
I20250625 13:52:09.579658 1168 rpcz_store.cc:276] 0625 13:52:06.632339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.633601 (+ 1262us) service_pool.cc:225] Handling call
0625 13:52:09.579499 (+2945898us) ts_tablet_manager.cc:1145] Deleting tablet 74884bf313724d678d3b5f5869f09f97
0625 13:52:09.579507 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.579573 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2945769}
I20250625 13:52:09.579931 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 734) took 2901 ms. Trace:
I20250625 13:52:09.579993 1168 rpcz_store.cc:276] 0625 13:52:06.678444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.678498 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:09.579852 (+2901354us) ts_tablet_manager.cc:1145] Deleting tablet 6e8b9b732f8948d2b4c1a81a1de4489a
0625 13:52:09.579860 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.579919 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2901151}
I20250625 13:52:09.580251 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 735) took 2896 ms. Trace:
I20250625 13:52:09.580317 1168 rpcz_store.cc:276] 0625 13:52:06.684051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.684110 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:09.580177 (+2896067us) ts_tablet_manager.cc:1145] Deleting tablet cbe4046742b34f1ab0b1610b5d6ee01e
0625 13:52:09.580184 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.580240 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2895872}
I20250625 13:52:09.580598 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 736) took 2895 ms. Trace:
I20250625 13:52:09.580674 1168 rpcz_store.cc:276] 0625 13:52:06.685304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.685350 (+ 46us) service_pool.cc:225] Handling call
0625 13:52:09.580519 (+2895169us) ts_tablet_manager.cc:1145] Deleting tablet 5d478ac74f4f46c6a29a6ac031152498
0625 13:52:09.580526 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.580585 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2894999}
I20250625 13:52:09.580966 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 737) took 2696 ms. Trace:
I20250625 13:52:09.581035 1168 rpcz_store.cc:276] 0625 13:52:06.884891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.884958 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:09.580872 (+2695914us) ts_tablet_manager.cc:1145] Deleting tablet 0998ad87fcc145999f6430e0de7b1304
0625 13:52:09.580879 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.580940 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2695696}
W20250625 13:52:09.580353 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet b14d00c93a964edd815122e3723f89e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b14d00c93a964edd815122e3723f89e7 already in progress: creating tablet
W20250625 13:52:09.582075 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8e2bc4df39674dc3be0cd32c68f1ebca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2bc4df39674dc3be0cd32c68f1ebca already in progress: creating tablet
W20250625 13:52:09.582330 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c431abffd05c4136b7510c60eee2a8b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c431abffd05c4136b7510c60eee2a8b1 already in progress: creating tablet
W20250625 13:52:09.582561 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 68a7712455694a1cbead413230620827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68a7712455694a1cbead413230620827 already in progress: creating tablet
W20250625 13:52:09.582913 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c366da4fb138453386d97fba0f5c91be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c366da4fb138453386d97fba0f5c91be already in progress: creating tablet
I20250625 13:52:09.581813 1165 ts_tablet_manager.cc:1918] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250625 13:52:09.583184 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 74884bf313724d678d3b5f5869f09f97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74884bf313724d678d3b5f5869f09f97 already in progress: creating tablet
W20250625 13:52:09.583456 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6e8b9b732f8948d2b4c1a81a1de4489a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e8b9b732f8948d2b4c1a81a1de4489a already in progress: creating tablet
I20250625 13:52:09.583629 1165 log.cc:1199] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/6dd84b8afa2548fe8ce5925f0bf806f0
W20250625 13:52:09.583756 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 5d478ac74f4f46c6a29a6ac031152498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d478ac74f4f46c6a29a6ac031152498 already in progress: creating tablet
W20250625 13:52:09.583971 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 0998ad87fcc145999f6430e0de7b1304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0998ad87fcc145999f6430e0de7b1304 already in progress: creating tablet
I20250625 13:52:09.586829 1171 tablet_bootstrap.cc:492] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.588392 1171 tablet_bootstrap.cc:654] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.590093 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 29a542de77674e33b70647ffd9475fbf
I20250625 13:52:09.591419 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 39da3ebbe55f4c4fab4f8af444657b9b
I20250625 13:52:09.591883 597 catalog_manager.cc:6652] ReplaceTablet: tablet 39da3ebbe55f4c4fab4f8af444657b9b deleted and replaced by tablet 9265e1a660a84ef19863ccdbad72033d
I20250625 13:52:09.592633 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 738) took 2695 ms. Trace:
I20250625 13:52:09.592715 1168 rpcz_store.cc:276] 0625 13:52:06.896796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.896986 (+ 190us) service_pool.cc:225] Handling call
0625 13:52:09.581204 (+2684218us) ts_tablet_manager.cc:1145] Deleting tablet 474356bf56b94681a8dc9071c0d8e79a
0625 13:52:09.581210 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.592615 (+ 11405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2684042}
I20250625 13:52:09.593283 1842 raft_consensus.cc:491] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.593416 1842 raft_consensus.cc:513] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.594173 1842 leader_election.cc:290] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:09.594210 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 29a542de77674e33b70647ffd9475fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.594866 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df1ed922750f45eab02d1fc0c26af75e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.595263 849 leader_election.cc:343] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.595546 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 29a542de77674e33b70647ffd9475fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.595731 1170 tablet_bootstrap.cc:492] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:09.595849 1170 ts_tablet_manager.cc:1397] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.002s
I20250625 13:52:09.596175 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 39da3ebbe55f4c4fab4f8af444657b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 9265e1a660a84ef19863ccdbad72033d) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.597002 1170 raft_consensus.cc:357] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.597352 1170 raft_consensus.cc:383] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.597373 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3e60792b6796458c857415e258ece1c9
I20250625 13:52:09.597465 1170 raft_consensus.cc:738] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.597829 1170 consensus_queue.cc:260] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:09.598029 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 739) took 2649 ms. Trace:
I20250625 13:52:09.598100 1168 rpcz_store.cc:276] 0625 13:52:06.948340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.948742 (+ 402us) service_pool.cc:225] Handling call
0625 13:52:09.597916 (+2649174us) ts_tablet_manager.cc:1145] Deleting tablet 155737a8d5604eb69524c4d6bea61550
0625 13:52:09.597930 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.598014 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2648954}
I20250625 13:52:09.598519 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 740) took 2603 ms. Trace:
I20250625 13:52:09.598587 1168 rpcz_store.cc:276] 0625 13:52:06.995489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:06.995574 (+ 85us) service_pool.cc:225] Handling call
0625 13:52:09.598426 (+2602852us) ts_tablet_manager.cc:1145] Deleting tablet 474356bf56b94681a8dc9071c0d8e79a
0625 13:52:09.598434 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.598506 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2602672}
I20250625 13:52:09.598994 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 747) took 2480 ms. Trace:
I20250625 13:52:09.599064 1168 rpcz_store.cc:276] 0625 13:52:07.118638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.118693 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:09.598905 (+2480212us) ts_tablet_manager.cc:1145] Deleting tablet 5788c3143f8c4368a0a765a1827134a9
0625 13:52:09.598913 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.598979 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2479987}
I20250625 13:52:09.599649 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 750) took 2412 ms. Trace:
I20250625 13:52:09.599721 1168 rpcz_store.cc:276] 0625 13:52:07.187307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.187388 (+ 81us) service_pool.cc:225] Handling call
0625 13:52:09.599560 (+2412172us) ts_tablet_manager.cc:1145] Deleting tablet bdbf586b82194a24a2259cbeb62d615c
0625 13:52:09.599567 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.599634 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2411982}
I20250625 13:52:09.600111 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 763) took 2310 ms. Trace:
I20250625 13:52:09.600175 1168 rpcz_store.cc:276] 0625 13:52:07.289603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.289664 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:09.600029 (+2310365us) ts_tablet_manager.cc:1145] Deleting tablet f1d647c2504c494d8b60c751126ee825
0625 13:52:09.600037 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.600097 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310190}
I20250625 13:52:09.601202 1171 tablet_bootstrap.cc:492] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.601317 1171 ts_tablet_manager.cc:1397] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250625 13:52:09.602100 1171 raft_consensus.cc:357] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.602259 1171 raft_consensus.cc:383] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.602310 1171 raft_consensus.cc:738] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.602535 1171 consensus_queue.cc:260] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.602661 1170 ts_tablet_manager.cc:1428] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250625 13:52:09.603117 1171 ts_tablet_manager.cc:1428] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:09.603405 1171 tablet_bootstrap.cc:492] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.604974 1171 tablet_bootstrap.cc:654] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.606145 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 765) took 2315 ms. Trace:
I20250625 13:52:09.606225 1168 rpcz_store.cc:276] 0625 13:52:07.291052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.291131 (+ 79us) service_pool.cc:225] Handling call
0625 13:52:09.606014 (+2314883us) ts_tablet_manager.cc:1145] Deleting tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76
0625 13:52:09.606024 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.606125 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2314703}
I20250625 13:52:09.608286 2147 raft_consensus.cc:491] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.608448 2147 raft_consensus.cc:513] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.609651 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6b0fc9a5299d475aa3ac845d06953e17" } new_table_name: "default.Table_6fadcfd061ab4fbdbd4bcbef13df8910" modify_external_catalogs: true
I20250625 13:52:09.611649 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3cac432f7e1f4bf28577a482f3cab0df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.611747 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 474356bf56b94681a8dc9071c0d8e79a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 474356bf56b94681a8dc9071c0d8e79a already in progress: creating tablet
W20250625 13:52:09.612210 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cac432f7e1f4bf28577a482f3cab0df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:09.612300 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 155737a8d5604eb69524c4d6bea61550 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 155737a8d5604eb69524c4d6bea61550 already in progress: creating tablet
W20250625 13:52:09.612780 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 474356bf56b94681a8dc9071c0d8e79a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 474356bf56b94681a8dc9071c0d8e79a already in progress: creating tablet
W20250625 13:52:09.613123 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 5788c3143f8c4368a0a765a1827134a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5788c3143f8c4368a0a765a1827134a9 already in progress: creating tablet
I20250625 13:52:09.613437 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_be523f2a377f4b8ba2de9c710ea57a9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.613508 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bdbf586b82194a24a2259cbeb62d615c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdbf586b82194a24a2259cbeb62d615c already in progress: creating tablet
W20250625 13:52:09.613596 849 consensus_peers.cc:487] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:09.613781 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 39da3ebbe55f4c4fab4f8af444657b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 9265e1a660a84ef19863ccdbad72033d) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.614028 1168 tablet_replica.cc:331] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.614226 1168 raft_consensus.cc:2241] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.614355 1168 raft_consensus.cc:2270] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:09.613860 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet f1d647c2504c494d8b60c751126ee825 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1d647c2504c494d8b60c751126ee825 already in progress: creating tablet
W20250625 13:52:09.614511 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be523f2a377f4b8ba2de9c710ea57a9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.614934 1168 ts_tablet_manager.cc:1905] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:09.614913 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a03a4fd4f444a4ab4a6a0ebb8321d76 already in progress: creating tablet
I20250625 13:52:09.617559 595 catalog_manager.cc:5582] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3). New cstate: current_term: 1 leader_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:09.618423 595 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:09.618763 595 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:09.619657 1170 tablet_bootstrap.cc:492] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:09.621176 1170 tablet_bootstrap.cc:654] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.622159 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4d11b567f2c14b62a446981c6ad894d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.622915 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d11b567f2c14b62a446981c6ad894d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.623631 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c9b44fa6c36f4b0fab24634ea2235568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.624197 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9b44fa6c36f4b0fab24634ea2235568 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.626999 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3a79d0adb2204bcaa209ad1de0fa64fc" } modify_external_catalogs: true
I20250625 13:52:09.627214 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3a79d0adb2204bcaa209ad1de0fa64fc" } modify_external_catalogs: true
I20250625 13:52:09.628719 1842 raft_consensus.cc:491] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.628896 1842 raft_consensus.cc:513] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:09.629324 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 29a542de77674e33b70647ffd9475fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29a542de77674e33b70647ffd9475fbf already in progress: creating tablet
I20250625 13:52:09.629707 1842 leader_election.cc:290] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:09.630513 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b129117db064a15b27e82439637cf22" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:09.630738 1046 raft_consensus.cc:2466] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:09.630738 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 39da3ebbe55f4c4fab4f8af444657b9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39da3ebbe55f4c4fab4f8af444657b9b already in progress: creating tablet
I20250625 13:52:09.631460 849 leader_election.cc:304] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:09.632215 1842 raft_consensus.cc:2802] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:09.632324 1842 raft_consensus.cc:491] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:09.632404 1842 raft_consensus.cc:3058] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:09.632545 1168 ts_tablet_manager.cc:1918] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.632659 1168 log.cc:1199] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/abd6f9121a9d4b39adde1b880a875e23
I20250625 13:52:09.632968 1168 ts_tablet_manager.cc:1939] T abd6f9121a9d4b39adde1b880a875e23 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.633730 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 767) took 2336 ms. Trace:
I20250625 13:52:09.635259 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1e884456a1d24e429f5e791a07f1014f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:09.635725 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e60792b6796458c857415e258ece1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.635805 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e60792b6796458c857415e258ece1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:09.635979 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e884456a1d24e429f5e791a07f1014f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.636911 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 067272869c5547648c90f83a34a036aa with delete_type TABLET_DATA_DELETED (replaced by tablet 6340de862fd04c12bb7f85557cfe53d2) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.638042 881 tablet_service.cc:1515] Processing DeleteTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.633821 1168 rpcz_store.cc:276] 0625 13:52:07.297372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.297421 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:09.608577 (+2311156us) ts_tablet_manager.cc:1145] Deleting tablet abd6f9121a9d4b39adde1b880a875e23
0625 13:52:09.613988 (+ 5411us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.622371 (+ 8383us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.632437 (+ 10066us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.633712 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310894}
I20250625 13:52:09.638813 1168 tablet_replica.cc:331] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.638991 1168 raft_consensus.cc:2241] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.639137 1168 raft_consensus.cc:2270] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.639748 1168 ts_tablet_manager.cc:1905] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.641280 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet abd6f9121a9d4b39adde1b880a875e23 (table default.Table_5d5a62a1608c45929c8fa1c853a06a27 [id=41e7cd4d77014eb0b0f1c2336c32bf1a]) successfully deleted
W20250625 13:52:09.645046 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3e60792b6796458c857415e258ece1c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e60792b6796458c857415e258ece1c9 already in progress: creating tablet
I20250625 13:52:09.648640 1842 raft_consensus.cc:513] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.650243 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b129117db064a15b27e82439637cf22" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:09.650488 1046 raft_consensus.cc:3058] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:09.653250 1168 ts_tablet_manager.cc:1918] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.653373 1168 log.cc:1199] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/2bbd509e3dfb45d98314c6afe37677f6
I20250625 13:52:09.653697 1168 ts_tablet_manager.cc:1939] T 2bbd509e3dfb45d98314c6afe37677f6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.654167 1046 raft_consensus.cc:2466] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:09.654548 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 768) took 2357 ms. Trace:
I20250625 13:52:09.654659 1168 rpcz_store.cc:276] 0625 13:52:07.297546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.297595 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:09.638748 (+2341153us) ts_tablet_manager.cc:1145] Deleting tablet 2bbd509e3dfb45d98314c6afe37677f6
0625 13:52:09.638761 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.649747 (+ 10986us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.653139 (+ 3392us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.654526 (+ 1387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2341007}
I20250625 13:52:09.654821 849 leader_election.cc:304] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:09.655212 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 2bbd509e3dfb45d98314c6afe37677f6 (table default.Table_683f20e141ca4122bee28d4e86aece20 [id=40f54158a1c84f5988762e7a44e3eab1]) successfully deleted
I20250625 13:52:09.658973 1842 leader_election.cc:290] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:09.659490 1842 raft_consensus.cc:2802] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:09.659907 1842 raft_consensus.cc:695] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:09.660882 1842 consensus_queue.cc:237] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.669088 1165 ts_tablet_manager.cc:1939] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:09.669099 1170 tablet_bootstrap.cc:492] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:09.669245 1170 ts_tablet_manager.cc:1397] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.004s
I20250625 13:52:09.669993 1170 raft_consensus.cc:357] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.670149 1170 raft_consensus.cc:383] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.670197 1170 raft_consensus.cc:738] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.670437 1170 consensus_queue.cc:260] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.673215 754 tablet_service.cc:1468] Processing CreateTablet for tablet bf962c47820f45e79e0e70f9208a17ff (DEFAULT_TABLE table=default.Table_5bc830734d3046dcafc7d1ccc0c5c561 [id=7b3e78773eaa40e6b13090408bb266c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.673446 1170 ts_tablet_manager.cc:1428] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:09.673525 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e4738b2e204bb1bc53908429537d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.673748 1170 tablet_bootstrap.cc:492] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:09.674669 881 tablet_service.cc:1468] Processing CreateTablet for tablet 4181bbb4af9c4141b587968cb7ef8251 (DEFAULT_TABLE table=default.Table_2bea8ee8d6074d3785dc33d166fc3485 [id=6c81574e23834914a987a9c4003de66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.675071 1170 tablet_bootstrap.cc:654] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.675096 877 tablet_service.cc:1468] Processing CreateTablet for tablet 1cc40b04f7f841888c142451e7df70d2 (DEFAULT_TABLE table=default.Table_cc354854cd8f4c13b5d9f37728037992 [id=d80d2502edae4af2af7d6fe32d1df3a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.675539 892 tablet_service.cc:1468] Processing CreateTablet for tablet d6710249ce844de2a9e8b63ae65de9de (DEFAULT_TABLE table=default.Table_ce4451ff102a4c22ba9539daffd65aa6 [id=4148344340d545ae8f071f8292500291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.675894 888 tablet_service.cc:1468] Processing CreateTablet for tablet a2ebcb65c49746f3ba9aa905e657c107 (DEFAULT_TABLE table=default.Table_73b9f78abc404d3c98a2a57d96a30f84 [id=5ad39b08aadb4b91b2d027bd2a0e5fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.661170 1168 tablet_replica.cc:331] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.676235 894 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9 (DEFAULT_TABLE table=default.Table_0b1631a468c34b49a62449290f63ea32 [id=116969aa3f7a4ef29b7007b14e1deddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.676355 1168 raft_consensus.cc:2241] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.676533 1168 raft_consensus.cc:2270] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.676631 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 4181bbb4af9c4141b587968cb7ef8251 (DEFAULT_TABLE table=default.Table_2bea8ee8d6074d3785dc33d166fc3485 [id=6c81574e23834914a987a9c4003de66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.676997 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4181bbb4af9c4141b587968cb7ef8251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.677047 1168 ts_tablet_manager.cc:1905] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.679467 1020 tablet_service.cc:1468] Processing CreateTablet for tablet d6710249ce844de2a9e8b63ae65de9de (DEFAULT_TABLE table=default.Table_ce4451ff102a4c22ba9539daffd65aa6 [id=4148344340d545ae8f071f8292500291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.679564 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 1cc40b04f7f841888c142451e7df70d2 (DEFAULT_TABLE table=default.Table_cc354854cd8f4c13b5d9f37728037992 [id=d80d2502edae4af2af7d6fe32d1df3a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.679880 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6710249ce844de2a9e8b63ae65de9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.680063 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9 (DEFAULT_TABLE table=default.Table_0b1631a468c34b49a62449290f63ea32 [id=116969aa3f7a4ef29b7007b14e1deddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.680359 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.683583 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 539daaa9df12499bb4ffb747bd3cec1c (DEFAULT_TABLE table=default.Table_ae346c0db6734c4684eb50013dda4b4a [id=1cd3f55ac4944dc2a43b80265ab9f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.683962 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539daaa9df12499bb4ffb747bd3cec1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.691468 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc40b04f7f841888c142451e7df70d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.693152 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4181bbb4af9c4141b587968cb7ef8251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.695881 2147 leader_election.cc:290] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.696022 2147 raft_consensus.cc:491] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.696117 2147 raft_consensus.cc:513] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.696694 2147 leader_election.cc:290] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.696965 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc40b04f7f841888c142451e7df70d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.698676 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf962c47820f45e79e0e70f9208a17ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.699196 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a29708fb688a4a65b84ae2564a040b4c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:09.699288 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c12f55ed5b4425aea1bf45cdc0658e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:09.700139 711 leader_election.cc:343] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:09.700309 711 leader_election.cc:343] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.700788 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6710249ce844de2a9e8b63ae65de9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.700969 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e60792b6796458c857415e258ece1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.702996 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 39da3ebbe55f4c4fab4f8af444657b9b with delete_type TABLET_DATA_DELETED (replaced by tablet 9265e1a660a84ef19863ccdbad72033d) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.703512 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ebcb65c49746f3ba9aa905e657c107. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.707155 1168 ts_tablet_manager.cc:1918] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.707255 1168 log.cc:1199] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ecb9a98e402b4da8a9c2bd8317cef532
I20250625 13:52:09.707513 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.707629 1168 ts_tablet_manager.cc:1939] T ecb9a98e402b4da8a9c2bd8317cef532 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.708349 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 766) took 2411 ms. Trace:
I20250625 13:52:09.708428 1168 rpcz_store.cc:276] 0625 13:52:07.297281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.297629 (+ 348us) service_pool.cc:225] Handling call
0625 13:52:09.661111 (+2363482us) ts_tablet_manager.cc:1145] Deleting tablet ecb9a98e402b4da8a9c2bd8317cef532
0625 13:52:09.661125 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.692528 (+ 31403us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.707071 (+ 14543us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.708326 (+ 1255us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13,"tablet-delete.queue_time_us":2363361}
I20250625 13:52:09.715903 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 769) took 2351 ms. Trace:
I20250625 13:52:09.715999 1168 rpcz_store.cc:276] 0625 13:52:07.364017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.364123 (+ 106us) service_pool.cc:225] Handling call
0625 13:52:09.715749 (+2351626us) ts_tablet_manager.cc:1145] Deleting tablet e99fe5ad690049029b8325dd38a11580
0625 13:52:09.715764 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.715880 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2351315}
I20250625 13:52:09.716307 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 770) took 2316 ms. Trace:
I20250625 13:52:09.716361 1168 rpcz_store.cc:276] 0625 13:52:07.399600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.399952 (+ 352us) service_pool.cc:225] Handling call
0625 13:52:09.716225 (+2316273us) ts_tablet_manager.cc:1145] Deleting tablet d2972d4366c143498f746fd62a039acd
0625 13:52:09.716232 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.716296 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2316019}
I20250625 13:52:09.716591 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 771) took 2316 ms. Trace:
I20250625 13:52:09.716657 1168 rpcz_store.cc:276] 0625 13:52:07.399857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.399913 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:09.716516 (+2316603us) ts_tablet_manager.cc:1145] Deleting tablet ef2c3757c0284aaca70ec317c05195d6
0625 13:52:09.716522 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.716579 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2316296}
I20250625 13:52:09.716904 1168 tablet_replica.cc:331] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.717109 1168 raft_consensus.cc:2241] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:09.717401 1168 pending_rounds.cc:70] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:09.717446 1168 pending_rounds.cc:77] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520627876208640 op_type: NO_OP noop_request { }
I20250625 13:52:09.717437 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet ecb9a98e402b4da8a9c2bd8317cef532 (table default.Table_cc7e074a01254cbb9272fd4ea7dba124 [id=d85d6e75f7e9420096456eac1ab144cc]) successfully deleted
I20250625 13:52:09.717523 1168 raft_consensus.cc:2887] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:09.717595 1168 raft_consensus.cc:2270] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.718345 896 tablet_service.cc:1468] Processing CreateTablet for tablet 539daaa9df12499bb4ffb747bd3cec1c (DEFAULT_TABLE table=default.Table_ae346c0db6734c4684eb50013dda4b4a [id=1cd3f55ac4944dc2a43b80265ab9f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.718557 750 tablet_service.cc:1468] Processing CreateTablet for tablet d6710249ce844de2a9e8b63ae65de9de (DEFAULT_TABLE table=default.Table_ce4451ff102a4c22ba9539daffd65aa6 [id=4148344340d545ae8f071f8292500291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.718715 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539daaa9df12499bb4ffb747bd3cec1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.718928 887 tablet_service.cc:1468] Processing CreateTablet for tablet 34fe02d7b37f4d99bbae082943223e35 (DEFAULT_TABLE table=default.Table_cea4a53bfa854372839e62197285c18c [id=1f771e558b7f44f1a12f33b823d1e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.719058 885 tablet_service.cc:1468] Processing CreateTablet for tablet f5a50381bcaa4c51b6f6f5aa47237115 (DEFAULT_TABLE table=default.Table_a55e69e3ed874f37bfe161cf46b4dfb5 [id=3b6f30942ccf4b3589758fb1463b0f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.719238 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34fe02d7b37f4d99bbae082943223e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.719187 889 tablet_service.cc:1468] Processing CreateTablet for tablet aadf5675a82b442bb9efc49d4e62ad01 (DEFAULT_TABLE table=default.Table_c3e15d16f1e74232b39b0c990c64bb6e [id=4139a2f9cfc84892b42c875c6fd66d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.719395 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a50381bcaa4c51b6f6f5aa47237115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.719637 879 tablet_service.cc:1468] Processing CreateTablet for tablet a27869d581594527b052cb6a6023ff2e (DEFAULT_TABLE table=default.Table_b23f837492294813bff5281ce11e1b9b [id=9aed5c5940bc413a89960858e6e4d94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.719952 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27869d581594527b052cb6a6023ff2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.720000 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1057bb98f947858471b86568d4898f (DEFAULT_TABLE table=default.Table_a47b9f65587049a1901b3490529d730f [id=8a22e272df104587b14d38dc8302cc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.720270 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1057bb98f947858471b86568d4898f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.723474 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6710249ce844de2a9e8b63ae65de9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.723558 878 tablet_service.cc:1468] Processing CreateTablet for tablet 5e381fd4d8034bd997b5e51abe93201d (DEFAULT_TABLE table=default.Table_8cae88538eb141a281ef6a3405a519aa [id=28a995512a2a4ba2aa8e243335f4d6e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.723879 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e381fd4d8034bd997b5e51abe93201d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.726178 742 tablet_service.cc:1468] Processing CreateTablet for tablet 4181bbb4af9c4141b587968cb7ef8251 (DEFAULT_TABLE table=default.Table_2bea8ee8d6074d3785dc33d166fc3485 [id=6c81574e23834914a987a9c4003de66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.726565 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4181bbb4af9c4141b587968cb7ef8251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.727555 886 tablet_service.cc:1468] Processing CreateTablet for tablet 0c9ad2aef1cc4748bfb90b7790f23bc8 (DEFAULT_TABLE table=default.Table_419b7a5baa6e4aed809e690190c9f73a [id=82d23273bec34b0681095a5d35965bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.727878 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9ad2aef1cc4748bfb90b7790f23bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.728653 741 tablet_service.cc:1468] Processing CreateTablet for tablet 1cc40b04f7f841888c142451e7df70d2 (DEFAULT_TABLE table=default.Table_cc354854cd8f4c13b5d9f37728037992 [id=d80d2502edae4af2af7d6fe32d1df3a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.728981 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cc40b04f7f841888c142451e7df70d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.730937 746 tablet_service.cc:1468] Processing CreateTablet for tablet 539daaa9df12499bb4ffb747bd3cec1c (DEFAULT_TABLE table=default.Table_ae346c0db6734c4684eb50013dda4b4a [id=1cd3f55ac4944dc2a43b80265ab9f7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.731237 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539daaa9df12499bb4ffb747bd3cec1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.731544 893 tablet_service.cc:1468] Processing CreateTablet for tablet 59e488ea19d24973a2ff11231622b63f (DEFAULT_TABLE table=default.Table_b0fa30878dd349899cc45f2600401269 [id=87b410edb8684ec998afde5705cd6217]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.731709 744 tablet_service.cc:1468] Processing CreateTablet for tablet a2ebcb65c49746f3ba9aa905e657c107 (DEFAULT_TABLE table=default.Table_73b9f78abc404d3c98a2a57d96a30f84 [id=5ad39b08aadb4b91b2d027bd2a0e5fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.731894 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e488ea19d24973a2ff11231622b63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.732065 757 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9 (DEFAULT_TABLE table=default.Table_0b1631a468c34b49a62449290f63ea32 [id=116969aa3f7a4ef29b7007b14e1deddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.732319 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.734148 1168 ts_tablet_manager.cc:1905] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.734817 739 tablet_service.cc:1468] Processing CreateTablet for tablet aadf5675a82b442bb9efc49d4e62ad01 (DEFAULT_TABLE table=default.Table_c3e15d16f1e74232b39b0c990c64bb6e [id=4139a2f9cfc84892b42c875c6fd66d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.735611 880 tablet_service.cc:1468] Processing CreateTablet for tablet a6126389a3564d0090c3b99500998201 (DEFAULT_TABLE table=default.Table_791629af534b4515bda726fca5095041 [id=6cecf6eb341d4f5c91f83943bb8990aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.735931 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6126389a3564d0090c3b99500998201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.739598 884 tablet_service.cc:1468] Processing CreateTablet for tablet b1864559e28544809e4299a784553193 (DEFAULT_TABLE table=default.Table_1b641ce25e2e4126bd10bcd3d7f8386c [id=0b47db3407dd439694c0bc3451a1b370]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.742906 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadf5675a82b442bb9efc49d4e62ad01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.743302 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ebcb65c49746f3ba9aa905e657c107. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.743816 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadf5675a82b442bb9efc49d4e62ad01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.746078 1019 tablet_service.cc:1468] Processing CreateTablet for tablet aadf5675a82b442bb9efc49d4e62ad01 (DEFAULT_TABLE table=default.Table_c3e15d16f1e74232b39b0c990c64bb6e [id=4139a2f9cfc84892b42c875c6fd66d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.746462 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadf5675a82b442bb9efc49d4e62ad01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.747069 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a27869d581594527b052cb6a6023ff2e (DEFAULT_TABLE table=default.Table_b23f837492294813bff5281ce11e1b9b [id=9aed5c5940bc413a89960858e6e4d94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.747390 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27869d581594527b052cb6a6023ff2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.758914 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a2ebcb65c49746f3ba9aa905e657c107 (DEFAULT_TABLE table=default.Table_73b9f78abc404d3c98a2a57d96a30f84 [id=5ad39b08aadb4b91b2d027bd2a0e5fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.759389 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ebcb65c49746f3ba9aa905e657c107. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.760031 596 master_service.cc:948] ReplaceTablet: received request to replace tablet a89f9ffb5e4643f8915e9fd69dd3e842 from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:09.762342 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e99fe5ad690049029b8325dd38a11580 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e99fe5ad690049029b8325dd38a11580 already in progress: creating tablet
W20250625 13:52:09.762639 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d2972d4366c143498f746fd62a039acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2972d4366c143498f746fd62a039acd already in progress: creating tablet
W20250625 13:52:09.762900 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ef2c3757c0284aaca70ec317c05195d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef2c3757c0284aaca70ec317c05195d6 already in progress: creating tablet
I20250625 13:52:09.763979 887 tablet_service.cc:1468] Processing CreateTablet for tablet 83699efa521a44f092cb738002aa58e4 (DEFAULT_TABLE table=default.Table_6b6fcfe7007d45478310c91778e8fdff [id=8f33f671ccfa4a008124724bca904f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.764062 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:09.764120 896 tablet_service.cc:1468] Processing CreateTablet for tablet b610cb974e8c45eeb756278c193c5357 (DEFAULT_TABLE table=default.Table_50aa6fee302644afa6b847fd604555b3 [id=106eebd9055841afa03aa5f75c3b0e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.764323 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
I20250625 13:52:09.764514 890 tablet_service.cc:1468] Processing CreateTablet for tablet 546d4d9ec2d14111902bb3f8643473db (DEFAULT_TABLE table=default.Table_1882ec53ba0a4451a9fcf2b494242aea [id=b5d2fc3c14a14b7b9cbd08154cb74afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.764585 891 tablet_service.cc:1468] Processing CreateTablet for tablet e002de6d4d8a49c59a09bd311e5d843f (DEFAULT_TABLE table=default.Table_6180acf99695440fbee78730755721b3 [id=7e5724fc3c8d470b90809869be3074a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.764518 589 catalog_manager.cc:5582] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:09.764746 883 tablet_service.cc:1468] Processing CreateTablet for tablet e38a85c299f04d60a3c465571fd33971 (DEFAULT_TABLE table=default.Table_6755f077e9a0460d86f174be4593542a [id=87b9e4b0bf1341d8ab54ec5a0dbd89df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.764993 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
I20250625 13:52:09.765301 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:09.766084 882 tablet_service.cc:1468] Processing CreateTablet for tablet 70197c2655dd4e6ebf930fc4065f5bb8 (DEFAULT_TABLE table=default.Table_013cf6cca16c4490ad8457353b68afc9 [id=90c4dc9e241949a1b28b9505bae4f72a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:09.766973 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20c12f55ed5b4425aea1bf45cdc0658e consensus state has no leader
I20250625 13:52:09.767477 895 tablet_service.cc:1468] Processing CreateTablet for tablet 3713b85743d04649b27517412b3d9ea0 (DEFAULT_TABLE table=default.Table_e460144bc7964ddba939c069278e887d [id=0d8694804f4b4eee88736c5f396faf9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.767485 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83699efa521a44f092cb738002aa58e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.767903 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b610cb974e8c45eeb756278c193c5357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.768059 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546d4d9ec2d14111902bb3f8643473db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.768270 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1864559e28544809e4299a784553193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.768594 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002de6d4d8a49c59a09bd311e5d843f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.768955 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38a85c299f04d60a3c465571fd33971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.769210 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3713b85743d04649b27517412b3d9ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.771402 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70197c2655dd4e6ebf930fc4065f5bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.775841 2147 raft_consensus.cc:491] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.776007 2147 raft_consensus.cc:513] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.776763 2147 leader_election.cc:290] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.778187 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:09.780478 1168 ts_tablet_manager.cc:1918] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:09.780609 1168 log.cc:1199] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/1effed0fa10d409abcb31dbba54861d6
I20250625 13:52:09.780982 1168 ts_tablet_manager.cc:1939] T 1effed0fa10d409abcb31dbba54861d6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:09.781970 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "490c8902efeb46c99543747a8787c1c7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:09.782646 711 leader_election.cc:343] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.787900 885 tablet_service.cc:1468] Processing CreateTablet for tablet c74adb1322fc4074a5b29dc7b44ceea9 (DEFAULT_TABLE table=default.Table_a24126ce121148d29172d84ff249105f [id=891d3cdc59be4f10909737a10ab7f33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.788305 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74adb1322fc4074a5b29dc7b44ceea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.791903 879 tablet_service.cc:1468] Processing CreateTablet for tablet d2e8f4e3b45a4065b74485157f31247d (DEFAULT_TABLE table=default.Table_330a9a9673154b0fb697d5f2cc16a4ab [id=20cbf73d64ee43899a58b19cd4dad71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.796514 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2e8f4e3b45a4065b74485157f31247d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.799944 888 tablet_service.cc:1468] Processing CreateTablet for tablet 3de53cc4cc68418b9647066310b23262 (DEFAULT_TABLE table=default.Table_112725cbadf0437681278a5ff391dffe [id=93ee744481b34a49b7e6e981fb95cd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.800393 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de53cc4cc68418b9647066310b23262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.802647 740 tablet_service.cc:1468] Processing CreateTablet for tablet a27869d581594527b052cb6a6023ff2e (DEFAULT_TABLE table=default.Table_b23f837492294813bff5281ce11e1b9b [id=9aed5c5940bc413a89960858e6e4d94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.803035 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27869d581594527b052cb6a6023ff2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.807186 1821 raft_consensus.cc:491] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.807376 1821 raft_consensus.cc:513] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.807791 894 tablet_service.cc:1468] Processing CreateTablet for tablet bb9fcf91fa304a359213a3e5267a9cac (DEFAULT_TABLE table=default.Table_34d8b1f4040c4fc08b1f847c31060596 [id=60e74c6b3d24463b8ff09271747e0d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.808043 1821 leader_election.cc:290] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.808205 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9fcf91fa304a359213a3e5267a9cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.808905 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "063a793e0d7747668d2728fdc2160b5e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:09.809090 912 raft_consensus.cc:2466] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:09.809506 983 leader_election.cc:304] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:09.809939 1821 raft_consensus.cc:2802] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:09.810012 1821 raft_consensus.cc:491] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:09.810070 1821 raft_consensus.cc:3058] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:09.812773 1821 raft_consensus.cc:513] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.813935 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "063a793e0d7747668d2728fdc2160b5e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:09.814101 912 raft_consensus.cc:3058] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:09.814019 1821 leader_election.cc:290] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.816525 912 raft_consensus.cc:2466] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:09.816949 983 leader_election.cc:304] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:09.817328 1821 raft_consensus.cc:2802] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:09.817550 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5e381fd4d8034bd997b5e51abe93201d (DEFAULT_TABLE table=default.Table_8cae88538eb141a281ef6a3405a519aa [id=28a995512a2a4ba2aa8e243335f4d6e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.817935 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e381fd4d8034bd997b5e51abe93201d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.818259 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 29a542de77674e33b70647ffd9475fbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:09.818972 985 leader_election.cc:343] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 063a793e0d7747668d2728fdc2160b5e
W20250625 13:52:09.820081 581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 56746.5us
I20250625 13:52:09.821943 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 042359eacaa2451abc0fa6ed447a2905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:09.823485 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a29708fb688a4a65b84ae2564a040b4c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:09.826005 751 tablet_service.cc:1468] Processing CreateTablet for tablet 0c9ad2aef1cc4748bfb90b7790f23bc8 (DEFAULT_TABLE table=default.Table_419b7a5baa6e4aed809e690190c9f73a [id=82d23273bec34b0681095a5d35965bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.826373 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9ad2aef1cc4748bfb90b7790f23bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.792773 882 tablet_service.cc:1468] Processing CreateTablet for tablet 27d2144b03c94101aef0b216f601bf1c (DEFAULT_TABLE table=default.Table_7b7294492237439b955837e068dea004 [id=9aad29b9a6364c2191accad5f1604003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.829182 891 tablet_service.cc:1468] Processing CreateTablet for tablet 2770194711ec4a86bb1354517b095f60 (DEFAULT_TABLE table=default.Table_2c513d91226c4bf6aa6643cc25131331 [id=56c8e8e69f08428887e7f120dd4f405d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.829437 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d2144b03c94101aef0b216f601bf1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.792889 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 773) took 2369 ms. Trace:
I20250625 13:52:09.829618 1168 rpcz_store.cc:276] 0625 13:52:07.423207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.427205 (+ 3998us) service_pool.cc:225] Handling call
0625 13:52:09.716861 (+2289656us) ts_tablet_manager.cc:1145] Deleting tablet 1effed0fa10d409abcb31dbba54861d6
0625 13:52:09.716869 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.777697 (+ 60828us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.780403 (+ 2706us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:09.792870 (+ 12467us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11066,"tablet-delete.queue_time_us":2289352}
I20250625 13:52:09.830005 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2770194711ec4a86bb1354517b095f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.830087 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 774) took 2391 ms. Trace:
I20250625 13:52:09.830168 1168 rpcz_store.cc:276] 0625 13:52:07.438692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.438747 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:09.829946 (+2391199us) ts_tablet_manager.cc:1145] Deleting tablet e7bd4939714a4359846345ed0521701c
0625 13:52:09.829957 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.830068 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2391014}
I20250625 13:52:09.830343 2192 raft_consensus.cc:491] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.830469 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 775) took 2300 ms. Trace:
I20250625 13:52:09.830446 2192 raft_consensus.cc:513] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.830520 1168 rpcz_store.cc:276] 0625 13:52:07.529749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.529812 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:09.830376 (+2300564us) ts_tablet_manager.cc:1145] Deleting tablet d89b0ddcc14148faa1a6926fcbb027a8
0625 13:52:09.830384 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.830456 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2300404}
I20250625 13:52:09.830811 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 776) took 2120 ms. Trace:
I20250625 13:52:09.830871 1168 rpcz_store.cc:276] 0625 13:52:07.710156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.710565 (+ 409us) service_pool.cc:225] Handling call
0625 13:52:09.830715 (+2120150us) ts_tablet_manager.cc:1145] Deleting tablet f14db116c8184370a995e82e33813932
0625 13:52:09.830722 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.830797 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119958}
I20250625 13:52:09.831100 2192 leader_election.cc:290] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:09.834728 1171 tablet_bootstrap.cc:492] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.835216 1171 ts_tablet_manager.cc:1397] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.232s user 0.007s sys 0.015s
W20250625 13:52:09.839485 985 leader_election.cc:343] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 063a793e0d7747668d2728fdc2160b5e
I20250625 13:52:09.840214 1171 raft_consensus.cc:357] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.840380 1171 raft_consensus.cc:383] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.840440 1171 raft_consensus.cc:738] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.840665 1171 consensus_queue.cc:260] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.841131 1171 ts_tablet_manager.cc:1428] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:09.841408 1171 tablet_bootstrap.cc:492] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.842943 1171 tablet_bootstrap.cc:654] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.848791 749 tablet_service.cc:1468] Processing CreateTablet for tablet 59e488ea19d24973a2ff11231622b63f (DEFAULT_TABLE table=default.Table_b0fa30878dd349899cc45f2600401269 [id=87b410edb8684ec998afde5705cd6217]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.849449 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e488ea19d24973a2ff11231622b63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.854835 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "490c8902efeb46c99543747a8787c1c7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.855230 711 leader_election.cc:343] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.855283 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c12f55ed5b4425aea1bf45cdc0658e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:09.855370 711 leader_election.cc:304] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:09.856046 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a6126389a3564d0090c3b99500998201 (DEFAULT_TABLE table=default.Table_791629af534b4515bda726fca5095041 [id=6cecf6eb341d4f5c91f83943bb8990aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.856174 2147 raft_consensus.cc:2747] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.856451 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6126389a3564d0090c3b99500998201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.859584 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f5a50381bcaa4c51b6f6f5aa47237115 (DEFAULT_TABLE table=default.Table_a55e69e3ed874f37bfe161cf46b4dfb5 [id=3b6f30942ccf4b3589758fb1463b0f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.859959 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a50381bcaa4c51b6f6f5aa47237115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.861536 2147 raft_consensus.cc:491] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.861660 2147 raft_consensus.cc:513] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.862165 2147 leader_election.cc:290] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:09.862656 711 leader_election.cc:343] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.862748 711 leader_election.cc:304] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:09.862880 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "042359eacaa2451abc0fa6ed447a2905" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.863051 711 leader_election.cc:343] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.863121 711 leader_election.cc:304] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:09.863130 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:09.863507 2147 raft_consensus.cc:2747] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.863655 2147 raft_consensus.cc:2747] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.867095 745 tablet_service.cc:1468] Processing CreateTablet for tablet e002de6d4d8a49c59a09bd311e5d843f (DEFAULT_TABLE table=default.Table_6180acf99695440fbee78730755721b3 [id=7e5724fc3c8d470b90809869be3074a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.867097 744 tablet_service.cc:1468] Processing CreateTablet for tablet 546d4d9ec2d14111902bb3f8643473db (DEFAULT_TABLE table=default.Table_1882ec53ba0a4451a9fcf2b494242aea [id=b5d2fc3c14a14b7b9cbd08154cb74afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.867503 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002de6d4d8a49c59a09bd311e5d843f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.867820 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546d4d9ec2d14111902bb3f8643473db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.867729 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1057bb98f947858471b86568d4898f (DEFAULT_TABLE table=default.Table_a47b9f65587049a1901b3490529d730f [id=8a22e272df104587b14d38dc8302cc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.868062 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1057bb98f947858471b86568d4898f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.868019 1030 tablet_service.cc:1468] Processing CreateTablet for tablet b1864559e28544809e4299a784553193 (DEFAULT_TABLE table=default.Table_1b641ce25e2e4126bd10bcd3d7f8386c [id=0b47db3407dd439694c0bc3451a1b370]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.868321 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1864559e28544809e4299a784553193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.868525 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c74adb1322fc4074a5b29dc7b44ceea9 (DEFAULT_TABLE table=default.Table_a24126ce121148d29172d84ff249105f [id=891d3cdc59be4f10909737a10ab7f33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.868834 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74adb1322fc4074a5b29dc7b44ceea9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.872009 711 leader_election.cc:343] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.871943 1842 consensus_queue.cc:1035] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:09.875833 743 tablet_service.cc:1468] Processing CreateTablet for tablet e38a85c299f04d60a3c465571fd33971 (DEFAULT_TABLE table=default.Table_6755f077e9a0460d86f174be4593542a [id=87b9e4b0bf1341d8ab54ec5a0dbd89df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.876157 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38a85c299f04d60a3c465571fd33971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.876297 1166 tablet_bootstrap.cc:492] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:09.876852 1019 tablet_service.cc:1468] Processing CreateTablet for tablet bb9fcf91fa304a359213a3e5267a9cac (DEFAULT_TABLE table=default.Table_34d8b1f4040c4fc08b1f847c31060596 [id=60e74c6b3d24463b8ff09271747e0d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.877234 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9fcf91fa304a359213a3e5267a9cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.831118 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 777) took 2065 ms. Trace:
I20250625 13:52:09.878060 1168 rpcz_store.cc:276] 0625 13:52:07.765636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.765992 (+ 356us) service_pool.cc:225] Handling call
0625 13:52:09.831037 (+2065045us) ts_tablet_manager.cc:1145] Deleting tablet e2e216856d5a4a3f8cd3a3852ffe2c31
0625 13:52:09.831045 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.831105 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2064861}
I20250625 13:52:09.878441 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 778) took 2109 ms. Trace:
I20250625 13:52:09.878520 1168 rpcz_store.cc:276] 0625 13:52:07.768616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.768675 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:09.878342 (+2109667us) ts_tablet_manager.cc:1145] Deleting tablet c911d9ecbded417daf70e85c7cbcd52e
0625 13:52:09.878353 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.878427 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2109469}
W20250625 13:52:09.878821 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 44036.3us
I20250625 13:52:09.878885 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 780) took 1936 ms. Trace:
I20250625 13:52:09.878964 1168 rpcz_store.cc:276] 0625 13:52:07.942366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.942526 (+ 160us) service_pool.cc:225] Handling call
0625 13:52:09.878800 (+1936274us) ts_tablet_manager.cc:1145] Deleting tablet 6e0f2fa5d5614ad394e6e12ec62814d4
0625 13:52:09.878811 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.878868 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1902438}
I20250625 13:52:09.879254 1168 tablet_replica.cc:331] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:09.879477 1168 raft_consensus.cc:2241] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:09.879825 1168 pending_rounds.cc:70] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:09.879767 752 tablet_service.cc:1468] Processing CreateTablet for tablet 70197c2655dd4e6ebf930fc4065f5bb8 (DEFAULT_TABLE table=default.Table_013cf6cca16c4490ad8457353b68afc9 [id=90c4dc9e241949a1b28b9505bae4f72a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.879876 1168 pending_rounds.cc:77] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520626889814016 op_type: NO_OP noop_request { }
I20250625 13:52:09.879961 1168 raft_consensus.cc:2887] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:09.880043 1168 raft_consensus.cc:2270] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.879984 755 tablet_service.cc:1468] Processing CreateTablet for tablet d2e8f4e3b45a4065b74485157f31247d (DEFAULT_TABLE table=default.Table_330a9a9673154b0fb697d5f2cc16a4ab [id=20cbf73d64ee43899a58b19cd4dad71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.880100 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70197c2655dd4e6ebf930fc4065f5bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.880385 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3713b85743d04649b27517412b3d9ea0 (DEFAULT_TABLE table=default.Table_e460144bc7964ddba939c069278e887d [id=0d8694804f4b4eee88736c5f396faf9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.880589 1168 ts_tablet_manager.cc:1905] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.880738 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3713b85743d04649b27517412b3d9ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.881523 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 59e488ea19d24973a2ff11231622b63f (DEFAULT_TABLE table=default.Table_b0fa30878dd349899cc45f2600401269 [id=87b410edb8684ec998afde5705cd6217]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.882364 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 27d2144b03c94101aef0b216f601bf1c (DEFAULT_TABLE table=default.Table_7b7294492237439b955837e068dea004 [id=9aad29b9a6364c2191accad5f1604003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.882654 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d2144b03c94101aef0b216f601bf1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.883548 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2e8f4e3b45a4065b74485157f31247d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.883630 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 83699efa521a44f092cb738002aa58e4 (DEFAULT_TABLE table=default.Table_6b6fcfe7007d45478310c91778e8fdff [id=8f33f671ccfa4a008124724bca904f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.883998 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 546d4d9ec2d14111902bb3f8643473db (DEFAULT_TABLE table=default.Table_1882ec53ba0a4451a9fcf2b494242aea [id=b5d2fc3c14a14b7b9cbd08154cb74afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.884285 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546d4d9ec2d14111902bb3f8643473db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.884440 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e002de6d4d8a49c59a09bd311e5d843f (DEFAULT_TABLE table=default.Table_6180acf99695440fbee78730755721b3 [id=7e5724fc3c8d470b90809869be3074a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.884734 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e002de6d4d8a49c59a09bd311e5d843f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.884702 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 34fe02d7b37f4d99bbae082943223e35 (DEFAULT_TABLE table=default.Table_cea4a53bfa854372839e62197285c18c [id=1f771e558b7f44f1a12f33b823d1e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.885004 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34fe02d7b37f4d99bbae082943223e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.885027 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 3713b85743d04649b27517412b3d9ea0 (DEFAULT_TABLE table=default.Table_e460144bc7964ddba939c069278e887d [id=0d8694804f4b4eee88736c5f396faf9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.887290 748 tablet_service.cc:1468] Processing CreateTablet for tablet 27d2144b03c94101aef0b216f601bf1c (DEFAULT_TABLE table=default.Table_7b7294492237439b955837e068dea004 [id=9aad29b9a6364c2191accad5f1604003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.887665 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d2144b03c94101aef0b216f601bf1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.888520 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e488ea19d24973a2ff11231622b63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.885027 1012 tablet_service.cc:1468] Processing CreateTablet for tablet b610cb974e8c45eeb756278c193c5357 (DEFAULT_TABLE table=default.Table_50aa6fee302644afa6b847fd604555b3 [id=106eebd9055841afa03aa5f75c3b0e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.889348 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83699efa521a44f092cb738002aa58e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.889604 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b610cb974e8c45eeb756278c193c5357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.889986 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3713b85743d04649b27517412b3d9ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.891521 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 0c9ad2aef1cc4748bfb90b7790f23bc8 (DEFAULT_TABLE table=default.Table_419b7a5baa6e4aed809e690190c9f73a [id=82d23273bec34b0681095a5d35965bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.891507 758 tablet_service.cc:1468] Processing CreateTablet for tablet bb9fcf91fa304a359213a3e5267a9cac (DEFAULT_TABLE table=default.Table_34d8b1f4040c4fc08b1f847c31060596 [id=60e74c6b3d24463b8ff09271747e0d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.891830 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9ad2aef1cc4748bfb90b7790f23bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.891830 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9fcf91fa304a359213a3e5267a9cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.892242 741 tablet_service.cc:1468] Processing CreateTablet for tablet 3de53cc4cc68418b9647066310b23262 (DEFAULT_TABLE table=default.Table_112725cbadf0437681278a5ff391dffe [id=93ee744481b34a49b7e6e981fb95cd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.892603 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de53cc4cc68418b9647066310b23262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.893491 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d2e8f4e3b45a4065b74485157f31247d (DEFAULT_TABLE table=default.Table_330a9a9673154b0fb697d5f2cc16a4ab [id=20cbf73d64ee43899a58b19cd4dad71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.893687 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 1effed0fa10d409abcb31dbba54861d6 (table default.Table_ec42501f607c41ceb9d35f3dd2a38f34 [id=d2c6242f48434d3bb6c2dc01fa095a67]) successfully deleted
I20250625 13:52:09.893817 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2e8f4e3b45a4065b74485157f31247d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.893970 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e7bd4939714a4359846345ed0521701c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7bd4939714a4359846345ed0521701c already in progress: creating tablet
W20250625 13:52:09.894150 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d89b0ddcc14148faa1a6926fcbb027a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d89b0ddcc14148faa1a6926fcbb027a8 already in progress: creating tablet
W20250625 13:52:09.894344 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet f14db116c8184370a995e82e33813932 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f14db116c8184370a995e82e33813932 already in progress: creating tablet
W20250625 13:52:09.894698 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e2e216856d5a4a3f8cd3a3852ffe2c31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2e216856d5a4a3f8cd3a3852ffe2c31 already in progress: creating tablet
W20250625 13:52:09.894904 581 catalog_manager.cc:4895] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c911d9ecbded417daf70e85c7cbcd52e because the tablet was not found. No further retry: Not found: Tablet not found: c911d9ecbded417daf70e85c7cbcd52e
W20250625 13:52:09.895051 581 catalog_manager.cc:4895] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6e0f2fa5d5614ad394e6e12ec62814d4 because the tablet was not found. No further retry: Not found: Tablet not found: 6e0f2fa5d5614ad394e6e12ec62814d4
I20250625 13:52:09.895570 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 2770194711ec4a86bb1354517b095f60 (DEFAULT_TABLE table=default.Table_2c513d91226c4bf6aa6643cc25131331 [id=56c8e8e69f08428887e7f120dd4f405d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.895825 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "042359eacaa2451abc0fa6ed447a2905" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:09.896045 851 leader_election.cc:343] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: df1ed922750f45eab02d1fc0c26af75e
I20250625 13:52:09.896065 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 6dd84b8afa2548fe8ce5925f0bf806f0 (table default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]) successfully deleted
I20250625 13:52:09.896155 851 leader_election.cc:304] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:09.896497 1842 raft_consensus.cc:2747] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:09.897439 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3e60792b6796458c857415e258ece1c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e60792b6796458c857415e258ece1c9 already in progress: creating tablet
W20250625 13:52:09.897822 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 39da3ebbe55f4c4fab4f8af444657b9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39da3ebbe55f4c4fab4f8af444657b9b already in progress: creating tablet
I20250625 13:52:09.897869 757 tablet_service.cc:1468] Processing CreateTablet for tablet 34fe02d7b37f4d99bbae082943223e35 (DEFAULT_TABLE table=default.Table_cea4a53bfa854372839e62197285c18c [id=1f771e558b7f44f1a12f33b823d1e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.898175 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34fe02d7b37f4d99bbae082943223e35. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.898192 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 29a542de77674e33b70647ffd9475fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29a542de77674e33b70647ffd9475fbf already in progress: creating tablet
I20250625 13:52:09.899700 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e38a85c299f04d60a3c465571fd33971 (DEFAULT_TABLE table=default.Table_6755f077e9a0460d86f174be4593542a [id=87b9e4b0bf1341d8ab54ec5a0dbd89df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.899713 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 70197c2655dd4e6ebf930fc4065f5bb8 (DEFAULT_TABLE table=default.Table_013cf6cca16c4490ad8457353b68afc9 [id=90c4dc9e241949a1b28b9505bae4f72a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.900417 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2770194711ec4a86bb1354517b095f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.900763 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70197c2655dd4e6ebf930fc4065f5bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.901338 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdca99b734448118e0c42227c78e58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.901578 1710 tablet_replica.cc:331] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:09.901746 1710 raft_consensus.cc:2241] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.901755 756 tablet_service.cc:1468] Processing CreateTablet for tablet 2770194711ec4a86bb1354517b095f60 (DEFAULT_TABLE table=default.Table_2c513d91226c4bf6aa6643cc25131331 [id=56c8e8e69f08428887e7f120dd4f405d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.901891 1710 raft_consensus.cc:2270] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:09.902053 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2770194711ec4a86bb1354517b095f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.902185 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 042359eacaa2451abc0fa6ed447a2905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:09.903833 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 042359eacaa2451abc0fa6ed447a2905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.904095 1165 tablet_replica.cc:331] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:09.904232 1166 ts_tablet_manager.cc:1397] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.393s user 0.022s sys 0.039s
I20250625 13:52:09.904268 1165 raft_consensus.cc:2241] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:09.904407 1165 raft_consensus.cc:2270] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:09.897889 711 leader_election.cc:343] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.904721 711 leader_election.cc:304] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:09.904874 711 raft_consensus.cc:2684] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:52:09.904960 1165 ts_tablet_manager.cc:1905] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.909970 1710 ts_tablet_manager.cc:1905] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:09.910086 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38a85c299f04d60a3c465571fd33971. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.916174 747 tablet_service.cc:1468] Processing CreateTablet for tablet b610cb974e8c45eeb756278c193c5357 (DEFAULT_TABLE table=default.Table_50aa6fee302644afa6b847fd604555b3 [id=106eebd9055841afa03aa5f75c3b0e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.916388 1165 ts_tablet_manager.cc:1918] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.916498 1165 log.cc:1199] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:09.916644 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b610cb974e8c45eeb756278c193c5357. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.916802 1165 ts_tablet_manager.cc:1939] T 042359eacaa2451abc0fa6ed447a2905 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:09.919534 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3de53cc4cc68418b9647066310b23262 (DEFAULT_TABLE table=default.Table_112725cbadf0437681278a5ff391dffe [id=93ee744481b34a49b7e6e981fb95cd1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.919914 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de53cc4cc68418b9647066310b23262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.921108 1171 tablet_bootstrap.cc:492] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:09.921161 739 tablet_service.cc:1468] Processing CreateTablet for tablet c74adb1322fc4074a5b29dc7b44ceea9 (DEFAULT_TABLE table=default.Table_a24126ce121148d29172d84ff249105f [id=891d3cdc59be4f10909737a10ab7f33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.921288 1171 ts_tablet_manager.cc:1397] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20250625 13:52:09.921514 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74adb1322fc4074a5b29dc7b44ceea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.922021 1171 raft_consensus.cc:357] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.922200 1171 raft_consensus.cc:383] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.922262 1171 raft_consensus.cc:738] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.922432 1710 ts_tablet_manager.cc:1918] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:09.922549 1710 log.cc:1199] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/8fdca99b734448118e0c42227c78e58f
I20250625 13:52:09.922504 1171 consensus_queue.cc:260] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.923018 1171 ts_tablet_manager.cc:1428] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:09.923358 1171 tablet_bootstrap.cc:492] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:09.924954 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 5e381fd4d8034bd997b5e51abe93201d (DEFAULT_TABLE table=default.Table_8cae88538eb141a281ef6a3405a519aa [id=28a995512a2a4ba2aa8e243335f4d6e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.925328 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e381fd4d8034bd997b5e51abe93201d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.925453 1171 tablet_bootstrap.cc:654] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:09.932896 753 tablet_service.cc:1468] Processing CreateTablet for tablet f5a50381bcaa4c51b6f6f5aa47237115 (DEFAULT_TABLE table=default.Table_a55e69e3ed874f37bfe161cf46b4dfb5 [id=3b6f30942ccf4b3589758fb1463b0f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.933486 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a50381bcaa4c51b6f6f5aa47237115. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.935832 1710 ts_tablet_manager.cc:1939] T 8fdca99b734448118e0c42227c78e58f P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:09.945868 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 042359eacaa2451abc0fa6ed447a2905 (table default.Table_3c1061dac55a4e97ad139950a7c8f302 [id=426662edb72f4d1aa3e0aa1ee934e62d]) successfully deleted
I20250625 13:52:09.940402 2147 raft_consensus.cc:491] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:09.950865 2147 raft_consensus.cc:513] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.942655 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:09.951525 777 raft_consensus.cc:2466] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:09.951687 742 tablet_service.cc:1468] Processing CreateTablet for tablet a6126389a3564d0090c3b99500998201 (DEFAULT_TABLE table=default.Table_791629af534b4515bda726fca5095041 [id=6cecf6eb341d4f5c91f83943bb8990aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.936795 1166 raft_consensus.cc:357] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:09.952385 1166 raft_consensus.cc:383] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:09.952435 1166 raft_consensus.cc:738] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:09.939682 750 tablet_service.cc:1468] Processing CreateTablet for tablet 83699efa521a44f092cb738002aa58e4 (DEFAULT_TABLE table=default.Table_6b6fcfe7007d45478310c91778e8fdff [id=8f33f671ccfa4a008124724bca904f3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.952054 2147 leader_election.cc:290] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:09.952741 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83699efa521a44f092cb738002aa58e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.946942 746 tablet_service.cc:1468] Processing CreateTablet for tablet 4b1057bb98f947858471b86568d4898f (DEFAULT_TABLE table=default.Table_a47b9f65587049a1901b3490529d730f [id=8a22e272df104587b14d38dc8302cc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.953508 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ba1338d0562418eadabdb8dbcaedee2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:09.954073 711 leader_election.cc:343] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.954298 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ba1338d0562418eadabdb8dbcaedee2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:09.947197 754 tablet_service.cc:1468] Processing CreateTablet for tablet b1864559e28544809e4299a784553193 (DEFAULT_TABLE table=default.Table_1b641ce25e2e4126bd10bcd3d7f8386c [id=0b47db3407dd439694c0bc3451a1b370]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:09.954605 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1864559e28544809e4299a784553193. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:09.954859 711 leader_election.cc:343] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:09.955005 711 leader_election.cc:304] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:09.955374 2147 raft_consensus.cc:2747] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:09.959609 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b1057bb98f947858471b86568d4898f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.962883 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6126389a3564d0090c3b99500998201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:09.963786 1166 consensus_queue.cc:260] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:09.965178 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 93390.4us
I20250625 13:52:09.973961 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 753e4767d7b84515ad2c93437e46c65b
I20250625 13:52:09.977952 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4017f3e1f1534574b8241c6d417fec78" } new_table_name: "default.Table_5c84ad3ea67d4b2cbdc4fb4eb0ea2c32" modify_external_catalogs: true
I20250625 13:52:09.978789 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_247151d8380c493883228e0cb0586a4b" } new_table_name: "default.Table_57f80199b4354980b61485dafd3ebf7a" modify_external_catalogs: true
I20250625 13:52:09.980984 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ea94ac9a52d84ddda29e98fa67d38868"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.981539 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea94ac9a52d84ddda29e98fa67d38868 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.981868 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_015b4a77da244eaf9d34a281215e77c7" } new_table_name: "default.Table_b003a7ace5e74fafb5a936359d3cf7b5" modify_external_catalogs: true
I20250625 13:52:09.982213 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8b535d0341e94ac2b1ea25b603b076ea" } new_table_name: "default.Table_cbc60d9e09c5478391309eae3867f656" modify_external_catalogs: true
I20250625 13:52:09.982520 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_85881e844bd44e0e97ff03992f7359c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.983060 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85881e844bd44e0e97ff03992f7359c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.984217 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3774c3c6876d4c799d85694da32c4f74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:09.984730 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3774c3c6876d4c799d85694da32c4f74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:09.986214 2192 consensus_queue.cc:1035] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:52:09.986815 851 leader_election.cc:343] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 0b129117db064a15b27e82439637cf22
W20250625 13:52:09.987000 849 leader_election.cc:343] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:09.987134 851 leader_election.cc:343] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 0b129117db064a15b27e82439637cf22
I20250625 13:52:09.987448 2192 consensus_queue.cc:1035] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 13:52:09.989799 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 753e4767d7b84515ad2c93437e46c65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:09.993475 1842 consensus_queue.cc:1035] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:09.999245 1170 tablet_bootstrap.cc:492] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:09.999408 1170 ts_tablet_manager.cc:1397] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.326s user 0.002s sys 0.001s
W20250625 13:52:10.000643 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 753e4767d7b84515ad2c93437e46c65b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753e4767d7b84515ad2c93437e46c65b already in progress: creating tablet
I20250625 13:52:10.007217 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.007505 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 042359eacaa2451abc0fa6ed447a2905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.007588 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 490c8902efeb46c99543747a8787c1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.007753 747 tablet_service.cc:1515] Processing DeleteTablet for tablet a29708fb688a4a65b84ae2564a040b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 2c8cf5d6cd624a1797486c69bd2098fd) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.007923 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:10.009060 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c183b3ba6f145e4b4741008ca27588a consensus state has no leader
I20250625 13:52:10.009387 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a89f9ffb5e4643f8915e9fd69dd3e842
I20250625 13:52:10.009833 596 catalog_manager.cc:6652] ReplaceTablet: tablet a89f9ffb5e4643f8915e9fd69dd3e842 deleted and replaced by tablet 8d39a96a128943438ead46a70f47c798
I20250625 13:52:10.010708 758 tablet_service.cc:1515] Processing DeleteTablet for tablet a89f9ffb5e4643f8915e9fd69dd3e842 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39a96a128943438ead46a70f47c798) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.012805 1165 tablet_replica.cc:331] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:10.012879 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:10.012997 1165 raft_consensus.cc:2241] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:10.013067 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:10.013479 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:10.013757 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:10.015887 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:10.026137 1165 raft_consensus.cc:2270] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:10.026981 1165 ts_tablet_manager.cc:1905] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:10.029392 2192 consensus_queue.cc:1035] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:10.029418 1842 consensus_queue.cc:1035] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 13:52:10.030896 1166 ts_tablet_manager.cc:1428] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.127s user 0.001s sys 0.001s
I20250625 13:52:10.035614 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 8fdca99b734448118e0c42227c78e58f (table default.Table_cd4b8059082844b99439152f1be88d22 [id=d5037bf1fc844a148103da5d2ae95193]) successfully deleted
I20250625 13:52:10.044713 1166 tablet_bootstrap.cc:492] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:10.051265 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 042359eacaa2451abc0fa6ed447a2905 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 042359eacaa2451abc0fa6ed447a2905 already in progress: creating tablet
W20250625 13:52:10.060279 711 consensus_peers.cc:487] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:10.061828 1166 tablet_bootstrap.cc:654] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:10.063674 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_66dd8f78c67643d9828dee272954c648"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.064270 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66dd8f78c67643d9828dee272954c648 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.065912 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3ab9801efffc44f183b50b4081652e9f" } modify_external_catalogs: true
I20250625 13:52:10.066128 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3ab9801efffc44f183b50b4081652e9f" } modify_external_catalogs: true
I20250625 13:52:10.067330 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0f4d14b4d5d84cab89f1ce9fa6a60e62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.068073 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f4d14b4d5d84cab89f1ce9fa6a60e62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.068154 851 leader_election.cc:304] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:10.069491 2192 raft_consensus.cc:2802] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:10.069659 2192 raft_consensus.cc:491] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:10.069804 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_722a6167ceef49f3b3c899c6fae0f44a" } modify_external_catalogs: true
I20250625 13:52:10.069846 2192 raft_consensus.cc:3058] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:10.073120 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_722a6167ceef49f3b3c899c6fae0f44a" } modify_external_catalogs: true
I20250625 13:52:10.073141 2192 raft_consensus.cc:513] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.073945 2192 leader_election.cc:290] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:10.081167 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 753e4767d7b84515ad2c93437e46c65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.081560 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 753e4767d7b84515ad2c93437e46c65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.083424 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a89f9ffb5e4643f8915e9fd69dd3e842 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39a96a128943438ead46a70f47c798) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.083590 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet a89f9ffb5e4643f8915e9fd69dd3e842 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d39a96a128943438ead46a70f47c798) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.084259 1171 tablet_bootstrap.cc:492] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:10.084419 1171 ts_tablet_manager.cc:1397] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.161s user 0.011s sys 0.002s
I20250625 13:52:10.085253 1171 raft_consensus.cc:357] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.085448 1171 raft_consensus.cc:383] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.085551 1171 raft_consensus.cc:738] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.085701 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a0e81d4c904248eba858462bcf31c5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.088608 885 tablet_service.cc:1515] Processing DeleteTablet for tablet ac4d368347444846880954ced9651404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.088830 888 tablet_service.cc:1515] Processing DeleteTablet for tablet df1ed922750f45eab02d1fc0c26af75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.089134 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b129117db064a15b27e82439637cf22 with delete_type TABLET_DATA_DELETED (replaced by tablet 35aedb3b590d4917bac8fe0962d68fb3) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.089315 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.089396 882 tablet_service.cc:1515] Processing DeleteTablet for tablet b14d00c93a964edd815122e3723f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.089162 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fa801f7d98e4794b6ca439638f29e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:10.089798 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 753e4767d7b84515ad2c93437e46c65b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753e4767d7b84515ad2c93437e46c65b already in progress: creating tablet
W20250625 13:52:10.090059 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a89f9ffb5e4643f8915e9fd69dd3e842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a89f9ffb5e4643f8915e9fd69dd3e842 already in progress: creating tablet
I20250625 13:52:10.091799 1166 tablet_bootstrap.cc:492] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:10.091980 1166 ts_tablet_manager.cc:1397] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.001s
I20250625 13:52:10.093171 1165 ts_tablet_manager.cc:1918] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:10.093302 1165 log.cc:1199] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/7ba1338d0562418eadabdb8dbcaedee2
I20250625 13:52:10.093600 1165 ts_tablet_manager.cc:1939] T 7ba1338d0562418eadabdb8dbcaedee2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:10.094101 1166 raft_consensus.cc:357] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.094338 1166 raft_consensus.cc:383] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.094434 1166 raft_consensus.cc:738] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.094679 1166 consensus_queue.cc:260] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.095185 1166 ts_tablet_manager.cc:1428] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 13:52:10.095535 1166 tablet_bootstrap.cc:492] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:10.097272 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ac647b1d704142b94da8d506319466 consensus state has no leader
I20250625 13:52:10.097317 1166 tablet_bootstrap.cc:654] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:10.099581 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b29392cfeb14f46a60fd0731f220127 consensus state has no leader
I20250625 13:52:10.099900 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_105e4d24875c4713ae92e3577d1881dd" } modify_external_catalogs: true
I20250625 13:52:10.100072 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_105e4d24875c4713ae92e3577d1881dd" } modify_external_catalogs: true
W20250625 13:52:10.100628 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 consensus state has no leader
I20250625 13:52:10.100934 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_94eb551cf0ef413b871b98aff36fcdf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.101217 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:10.101433 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94eb551cf0ef413b871b98aff36fcdf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.102774 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_31d921b4f7ca4806bf89fd8bca5e4e4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.103252 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31d921b4f7ca4806bf89fd8bca5e4e4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.104871 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ab888f08b1984d8da2a92b8b396d4989" } new_table_name: "default.Table_7ae6d5c541cf45baaeb485b10d6c9f8a" modify_external_catalogs: true
I20250625 13:52:10.105865 596 master_service.cc:948] ReplaceTablet: received request to replace tablet abc44a41b30e4a53977ec81d6905612f from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:10.107867 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:10.108471 773 raft_consensus.cc:3058] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:10.109732 849 leader_election.cc:343] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.104929 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_96a404a50c0641bf98518a2c816eaefa" } modify_external_catalogs: true
I20250625 13:52:10.110261 2192 consensus_queue.cc:1035] T 04dd85668a47461e9e82f8b251468ad3 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:10.110401 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_96a404a50c0641bf98518a2c816eaefa" } modify_external_catalogs: true
I20250625 13:52:10.111158 2147 raft_consensus.cc:491] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.111292 2147 raft_consensus.cc:513] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.111920 2147 leader_election.cc:290] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:10.114554 2192 raft_consensus.cc:491] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.114748 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32143e2497ee4d1b8c2e7cbfc6160ef4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:10.114732 2192 raft_consensus.cc:513] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.115981 2192 leader_election.cc:290] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:10.115993 711 leader_election.cc:343] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.116444 1171 consensus_queue.cc:260] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.111289 773 raft_consensus.cc:2466] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:10.123445 2207 consensus_queue.cc:1035] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:10.129163 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 430c3bed360b4f32855a8adb99076608 with delete_type TABLET_DATA_DELETED (replaced by tablet 0f63e9037bd74f1eb39bd85aa820ec35) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.130018 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 2404e712e18146b68757bd77362c1358 with delete_type TABLET_DATA_DELETED (replaced by tablet 68831b9ad1c14f5e9c954cbd06094896) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.131911 1170 raft_consensus.cc:357] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.132133 1170 raft_consensus.cc:383] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.132193 1170 raft_consensus.cc:738] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.132421 1170 consensus_queue.cc:260] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.132979 1170 ts_tablet_manager.cc:1428] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.134s user 0.002s sys 0.000s
I20250625 13:52:10.133306 1170 tablet_bootstrap.cc:492] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:10.135914 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]). Expected version 2 got 0
I20250625 13:52:10.138393 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32143e2497ee4d1b8c2e7cbfc6160ef4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:10.139078 1165 tablet_replica.cc:331] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:10.139341 1165 raft_consensus.cc:2241] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:10.139573 1165 raft_consensus.cc:2270] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:10.140147 1165 ts_tablet_manager.cc:1905] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:10.138566 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 7ba1338d0562418eadabdb8dbcaedee2 (table default.Table_c7891a214410484c907ee024a3f40331 [id=21a2a642fff348518e79d4e4db1f32ad]) successfully deleted
I20250625 13:52:10.144687 851 leader_election.cc:304] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
W20250625 13:52:10.144847 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 042359eacaa2451abc0fa6ed447a2905 because the tablet was not found. No further retry: Not found: Tablet not found: 042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:10.145181 2210 raft_consensus.cc:2802] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:10.145392 2210 raft_consensus.cc:695] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:10.145813 2210 consensus_queue.cc:237] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:10.147431 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c561fcc70924ea5ab4ab2b569df215a consensus state has no leader
I20250625 13:52:10.147959 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac4d368347444846880954ced9651404" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:10.149171 711 leader_election.cc:343] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.149333 711 leader_election.cc:304] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:10.149602 2195 raft_consensus.cc:2747] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:10.155962 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (table default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916]). Expected version 5 got 0
W20250625 13:52:10.159575 851 leader_election.cc:343] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ac4d368347444846880954ced9651404
I20250625 13:52:10.164294 754 tablet_service.cc:1468] Processing CreateTablet for tablet d35517bbf1cb4695ae46094dc883175f (DEFAULT_TABLE table=default.Table_be523f2a377f4b8ba2de9c710ea57a9c [id=d0a2fad4d84045bf9426b091b2f4ac9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.164824 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35517bbf1cb4695ae46094dc883175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.164911 1170 tablet_bootstrap.cc:654] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:10.169140 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0012e95ff4924723a6e6aaf6b85da438
I20250625 13:52:10.170011 2195 consensus_queue.cc:1035] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:10.171367 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 156da2a965c64d1090a4f9d4eafdec70
I20250625 13:52:10.172526 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 156da2a965c64d1090a4f9d4eafdec70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.177711 2199 raft_consensus.cc:491] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.177887 2199 raft_consensus.cc:513] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.178535 2199 leader_election.cc:290] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:10.182035 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 0012e95ff4924723a6e6aaf6b85da438 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.182947 1165 ts_tablet_manager.cc:1918] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:10.183032 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 0012e95ff4924723a6e6aaf6b85da438 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.183813 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0012e95ff4924723a6e6aaf6b85da438 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.184511 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 156da2a965c64d1090a4f9d4eafdec70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.183065 1165 log.cc:1199] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/490c8902efeb46c99543747a8787c1c7
I20250625 13:52:10.185353 1021 tablet_service.cc:1468] Processing CreateTablet for tablet d35517bbf1cb4695ae46094dc883175f (DEFAULT_TABLE table=default.Table_be523f2a377f4b8ba2de9c710ea57a9c [id=d0a2fad4d84045bf9426b091b2f4ac9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.185815 1165 ts_tablet_manager.cc:1939] T 490c8902efeb46c99543747a8787c1c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:10.185849 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35517bbf1cb4695ae46094dc883175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.186132 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7389669b9b714262b43171af80118da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.186635 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7389669b9b714262b43171af80118da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.187986 1165 tablet_replica.cc:331] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:10.187988 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_05a86981b085438083683deb71ee651d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.188232 1165 raft_consensus.cc:2241] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:10.188417 1165 raft_consensus.cc:2270] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:10.188608 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05a86981b085438083683deb71ee651d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.190212 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 490c8902efeb46c99543747a8787c1c7 (table default.Table_2a8634c60e08492faffd30a254d0dc11 [id=a0e4a71b13d94a69affc3cc9e0cb5066]) successfully deleted
I20250625 13:52:10.190685 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_065fd6832b3b458cbedf3713e9d234b3" } modify_external_catalogs: true
I20250625 13:52:10.190858 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_065fd6832b3b458cbedf3713e9d234b3" } modify_external_catalogs: true
I20250625 13:52:10.193338 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 156da2a965c64d1090a4f9d4eafdec70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.193743 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1ec436c40a764dbba4243d2522dcfb1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.194207 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0012e95ff4924723a6e6aaf6b85da438 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0012e95ff4924723a6e6aaf6b85da438 already in progress: creating tablet
W20250625 13:52:10.194487 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 156da2a965c64d1090a4f9d4eafdec70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156da2a965c64d1090a4f9d4eafdec70 already in progress: creating tablet
W20250625 13:52:10.194670 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ec436c40a764dbba4243d2522dcfb1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.197182 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet abc44a41b30e4a53977ec81d6905612f
I20250625 13:52:10.197602 596 catalog_manager.cc:6652] ReplaceTablet: tablet abc44a41b30e4a53977ec81d6905612f deleted and replaced by tablet d60f5d5746d44a8a8bb4a317d15f306b
I20250625 13:52:10.198365 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet af934858a4af4621bb55eb5a528fa14c
I20250625 13:52:10.199795 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3beb5dcb4f1e4ece821c83c3a52d93f5
I20250625 13:52:10.201886 741 tablet_service.cc:1515] Processing DeleteTablet for tablet af934858a4af4621bb55eb5a528fa14c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.202314 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.202527 753 tablet_service.cc:1515] Processing DeleteTablet for tablet abc44a41b30e4a53977ec81d6905612f with delete_type TABLET_DATA_DELETED (replaced by tablet d60f5d5746d44a8a8bb4a317d15f306b) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.204576 885 tablet_service.cc:1515] Processing DeleteTablet for tablet abc44a41b30e4a53977ec81d6905612f with delete_type TABLET_DATA_DELETED (replaced by tablet d60f5d5746d44a8a8bb4a317d15f306b) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.205802 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d5cb3b35a5c2483fa82226133a7c161d
I20250625 13:52:10.206030 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet abc44a41b30e4a53977ec81d6905612f with delete_type TABLET_DATA_DELETED (replaced by tablet d60f5d5746d44a8a8bb4a317d15f306b) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.208391 1165 ts_tablet_manager.cc:1905] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:10.209162 2231 tablet.cc:1722] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:10.210788 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_07265699458b42e8976c62a23fd2a135" } new_table_name: "default.Table_a5dde0fa6d7c471f983b3c8e01eabbcd" modify_external_catalogs: true
I20250625 13:52:10.213719 749 tablet_service.cc:1468] Processing CreateTablet for tablet 8285f51c022548fa9c73c8dd84b2b8c1 (DEFAULT_TABLE table=default.Table_1e884456a1d24e429f5e791a07f1014f [id=9361b4021f3a4024b7608b0a6996ae44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.214111 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8285f51c022548fa9c73c8dd84b2b8c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.208627 753 tablet_service.cc:1515] Processing DeleteTablet for tablet d5cb3b35a5c2483fa82226133a7c161d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.214859 2201 raft_consensus.cc:491] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:10.216567 851 leader_election.cc:343] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 430c3bed360b4f32855a8adb99076608
I20250625 13:52:10.216871 885 tablet_service.cc:1515] Processing DeleteTablet for tablet af934858a4af4621bb55eb5a528fa14c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.217631 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.218191 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d5cb3b35a5c2483fa82226133a7c161d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.219082 885 tablet_service.cc:1468] Processing CreateTablet for tablet d35517bbf1cb4695ae46094dc883175f (DEFAULT_TABLE table=default.Table_be523f2a377f4b8ba2de9c710ea57a9c [id=d0a2fad4d84045bf9426b091b2f4ac9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.219555 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d35517bbf1cb4695ae46094dc883175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.215268 2201 raft_consensus.cc:513] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.228185 2223 tablet.cc:1722] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 13:52:10.228436 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34371795c8094e73ab4e5dbeedb7aa2f consensus state has no leader
I20250625 13:52:10.230021 749 tablet_service.cc:1468] Processing CreateTablet for tablet 146ca3d1da24429e8487886e5e5efd6b (DEFAULT_TABLE table=default.Table_c9b44fa6c36f4b0fab24634ea2235568 [id=06e1a3dddf984b508ec13579bd9322b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.230418 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146ca3d1da24429e8487886e5e5efd6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.232455 1165 ts_tablet_manager.cc:1918] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:10.236851 589 catalog_manager.cc:5582] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:10.237433 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
W20250625 13:52:10.242772 849 leader_election.cc:343] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.242897 2199 raft_consensus.cc:491] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.243119 2199 raft_consensus.cc:513] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.242203 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9ec33dce871a442ea93577db109d42e6 (DEFAULT_TABLE table=default.Table_b45de936fca240bd8d8d63cac7007f5e [id=0af3ac93e56643548b3da06116a13b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.243854 2199 leader_election.cc:290] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:10.243002 2213 raft_consensus.cc:491] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.244441 2213 raft_consensus.cc:513] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.244701 739 tablet_service.cc:1468] Processing CreateTablet for tablet ed3f5984ecde4a81aaba0802b69a5ccc (DEFAULT_TABLE table=default.Table_3cac432f7e1f4bf28577a482f3cab0df [id=f8aba8cd9bdd478f9feb9ae521795239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.246008 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:10.246578 711 leader_election.cc:343] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:10.246925 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet abc44a41b30e4a53977ec81d6905612f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc44a41b30e4a53977ec81d6905612f already in progress: creating tablet
I20250625 13:52:10.247115 2213 leader_election.cc:290] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:10.247834 1165 log.cc:1199] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a29708fb688a4a65b84ae2564a040b4c
I20250625 13:52:10.248091 882 tablet_service.cc:1468] Processing CreateTablet for tablet 146ca3d1da24429e8487886e5e5efd6b (DEFAULT_TABLE table=default.Table_c9b44fa6c36f4b0fab24634ea2235568 [id=06e1a3dddf984b508ec13579bd9322b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.248360 1165 ts_tablet_manager.cc:1939] T a29708fb688a4a65b84ae2564a040b4c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:10.248510 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146ca3d1da24429e8487886e5e5efd6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.249331 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
I20250625 13:52:10.249528 1166 tablet_bootstrap.cc:492] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:10.249663 1166 ts_tablet_manager.cc:1397] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.154s user 0.005s sys 0.000s
I20250625 13:52:10.250200 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet dc75a0a8421b457089cc707d9eb6b76f (table default.Table_f618b45e019044ea946538ed3fa05471 [id=fa0246aff3f648538e6fddcb30f3a399]). Expected version 3 got 0
I20250625 13:52:10.250309 1166 raft_consensus.cc:357] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.250487 1166 raft_consensus.cc:383] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.250567 1166 raft_consensus.cc:738] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.250685 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:10.250818 1166 consensus_queue.cc:260] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.251561 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec33dce871a442ea93577db109d42e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.253386 749 tablet_service.cc:1468] Processing CreateTablet for tablet ea383a1ceddc461fb2531dbca714029e (DEFAULT_TABLE table=default.Table_d47cfb3324a443628070fd4437024a71 [id=cc661f7c95d442f881c5ce0ef5eec921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.253959 885 tablet_service.cc:1468] Processing CreateTablet for tablet 8285f51c022548fa9c73c8dd84b2b8c1 (DEFAULT_TABLE table=default.Table_1e884456a1d24e429f5e791a07f1014f [id=9361b4021f3a4024b7608b0a6996ae44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.254374 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8285f51c022548fa9c73c8dd84b2b8c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.254722 741 tablet_service.cc:1468] Processing CreateTablet for tablet 1068b65734bc4c29ba8b09a7a3d575b1 (DEFAULT_TABLE table=default.Table_338af88ef39143f0823658cf2b61c83b [id=c7b896b672834d3f8d4e698685034436]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:10.258324 851 leader_election.cc:343] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:10.259193 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
I20250625 13:52:10.261230 750 tablet_service.cc:1468] Processing CreateTablet for tablet 21ccd2404d954c6393476fdbcdc5d37a (DEFAULT_TABLE table=default.Table_458d788e23274f5d9f4044fe32959484 [id=c6ec184916a8470eaf500367cfe4e56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.261850 754 tablet_service.cc:1468] Processing CreateTablet for tablet 39e7c95bd30e403f9cd391f01ccec124 (DEFAULT_TABLE table=default.Table_47e85ba24d824e5eaf587f4dbc7c9d22 [id=3ee1e37050ad4e958797be261e5dc440]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.263319 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1a42bf5f0ad84a43a8de1a7a39138d0e (DEFAULT_TABLE table=default.Table_5d22cae7274d4d64811664493a162782 [id=7cd4907ca19e486d9d2390bc84a0018a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.081315 1168 ts_tablet_manager.cc:1918] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:10.264714 1168 log.cc:1199] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/36b151e200e1430c99ac194bc55881c1
I20250625 13:52:10.264827 758 tablet_service.cc:1468] Processing CreateTablet for tablet 776d546c733f4d1d8bbbdd64e23e9e62 (DEFAULT_TABLE table=default.Table_610e4cc494e244878a60d2382727f85c [id=c143510d38964470815db0340ebd2ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.265192 1168 ts_tablet_manager.cc:1939] T 36b151e200e1430c99ac194bc55881c1 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:10.272315 747 tablet_service.cc:1468] Processing CreateTablet for tablet 934976b12e674cce8d1e4f8fad60fc18 (DEFAULT_TABLE table=default.Table_717dce88b94246cab98733e4565a27b5 [id=214da52a59a148e3bee6ee3a40b85bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.272414 849 leader_election.cc:304] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:10.273015 2210 raft_consensus.cc:2747] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:10.274197 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
I20250625 13:52:10.275146 743 tablet_service.cc:1468] Processing CreateTablet for tablet c930ac853a304fbab4be6fb37df5f79a (DEFAULT_TABLE table=default.Table_2a88884ec2db4ae8bc4e94280d4c5022 [id=f8f92568dbf84291896ba4e79bbb6614]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.275964 748 tablet_service.cc:1468] Processing CreateTablet for tablet 7507d50e9f464043b8901491c57391d6 (DEFAULT_TABLE table=default.Table_893c6cc9fca542d0ab7c03d3f08207cc [id=6ddaa553802c431484cdd55c6615dcb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.277390 752 tablet_service.cc:1468] Processing CreateTablet for tablet 47c70919181d4c5392f65df221ec6348 (DEFAULT_TABLE table=default.Table_6e03f752b3f2463b84f66809236ba398 [id=40cad82310ae482fa8d283f5cc888d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.278014 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dc75a0a8421b457089cc707d9eb6b76f (table default.Table_f618b45e019044ea946538ed3fa05471 [id=fa0246aff3f648538e6fddcb30f3a399]). Expected version 3 got 0
I20250625 13:52:10.279263 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:10.279358 740 tablet_service.cc:1468] Processing CreateTablet for tablet b43341acdd754726ac0fbb06d27eec1c (DEFAULT_TABLE table=default.Table_e3936edb9d8e47d08189e8903287c572 [id=06c61eaf6373492e8a72ba7d0442c601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.279551 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:10.279968 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed3f5984ecde4a81aaba0802b69a5ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.280270 1165 tablet_replica.cc:331] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:10.280520 1165 raft_consensus.cc:2241] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:10.280675 1165 raft_consensus.cc:2270] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:10.251266 1166 ts_tablet_manager.cc:1428] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:10.281198 1165 ts_tablet_manager.cc:1905] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:10.281683 1166 tablet_bootstrap.cc:492] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:10.283052 1166 tablet_bootstrap.cc:654] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:10.287070 745 tablet_service.cc:1468] Processing CreateTablet for tablet 3562e8b8731a4d76bb1a36189f957e07 (DEFAULT_TABLE table=default.Table_d8054fa43c994aab98b1d5541d0be6f9 [id=82ac4f198e8242fc9a95787dc3331c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.287130 756 tablet_service.cc:1468] Processing CreateTablet for tablet d151560702a547acb01a353cf1fb27e3 (DEFAULT_TABLE table=default.Table_107ccce5d138433db4fb8351f69e40a0 [id=423c4d58d233464ebe4fcd7a7d426f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.287426 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea383a1ceddc461fb2531dbca714029e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.287518 757 tablet_service.cc:1468] Processing CreateTablet for tablet a1263945be894618a7f9946e7ca50efd (DEFAULT_TABLE table=default.Table_ac6acb4e2c0e4001927ffb4230ad2ecf [id=bc1d4ec693154bb5aa80709fed7d41e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.287678 751 tablet_service.cc:1468] Processing CreateTablet for tablet d0174345facc4216b6ddeb8f8c73790d (DEFAULT_TABLE table=default.Table_4d11b567f2c14b62a446981c6ad894d4 [id=665f17d9e587412197dc3304230fcc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.287962 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1068b65734bc4c29ba8b09a7a3d575b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.288353 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ccd2404d954c6393476fdbcdc5d37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.288703 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a42bf5f0ad84a43a8de1a7a39138d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.289042 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776d546c733f4d1d8bbbdd64e23e9e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.289403 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934976b12e674cce8d1e4f8fad60fc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.289775 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7507d50e9f464043b8901491c57391d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.290185 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c70919181d4c5392f65df221ec6348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.291507 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a29708fb688a4a65b84ae2564a040b4c (table default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]) successfully deleted
I20250625 13:52:10.292241 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e7c95bd30e403f9cd391f01ccec124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.292783 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43341acdd754726ac0fbb06d27eec1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.292768 2223 tablet.cc:1722] T dc75a0a8421b457089cc707d9eb6b76f P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:10.294001 888 tablet_service.cc:1468] Processing CreateTablet for tablet ed3f5984ecde4a81aaba0802b69a5ccc (DEFAULT_TABLE table=default.Table_3cac432f7e1f4bf28577a482f3cab0df [id=f8aba8cd9bdd478f9feb9ae521795239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.294345 894 tablet_service.cc:1468] Processing CreateTablet for tablet 9ec33dce871a442ea93577db109d42e6 (DEFAULT_TABLE table=default.Table_b45de936fca240bd8d8d63cac7007f5e [id=0af3ac93e56643548b3da06116a13b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.294485 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c930ac853a304fbab4be6fb37df5f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.295548 879 tablet_service.cc:1468] Processing CreateTablet for tablet ea383a1ceddc461fb2531dbca714029e (DEFAULT_TABLE table=default.Table_d47cfb3324a443628070fd4437024a71 [id=cc661f7c95d442f881c5ce0ef5eec921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.295548 887 tablet_service.cc:1468] Processing CreateTablet for tablet 1068b65734bc4c29ba8b09a7a3d575b1 (DEFAULT_TABLE table=default.Table_338af88ef39143f0823658cf2b61c83b [id=c7b896b672834d3f8d4e698685034436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.296046 896 tablet_service.cc:1468] Processing CreateTablet for tablet 39e7c95bd30e403f9cd391f01ccec124 (DEFAULT_TABLE table=default.Table_47e85ba24d824e5eaf587f4dbc7c9d22 [id=3ee1e37050ad4e958797be261e5dc440]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.296483 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3562e8b8731a4d76bb1a36189f957e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.296687 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1263945be894618a7f9946e7ca50efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.296854 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d151560702a547acb01a353cf1fb27e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.297365 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0174345facc4216b6ddeb8f8c73790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.298166 2231 tablet.cc:1722] T dc75a0a8421b457089cc707d9eb6b76f P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:10.299230 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9bc32b4aab234a4e8bab3910a4a3102b" } new_table_name: "default.Table_a61060462ad540c8b2315b33af732d56" modify_external_catalogs: true
I20250625 13:52:10.309731 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f4e7c3d0a0cb4e4a94d0a2e2e63610c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.309890 883 tablet_service.cc:1468] Processing CreateTablet for tablet 21ccd2404d954c6393476fdbcdc5d37a (DEFAULT_TABLE table=default.Table_458d788e23274f5d9f4044fe32959484 [id=c6ec184916a8470eaf500367cfe4e56f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:10.310338 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4e7c3d0a0cb4e4a94d0a2e2e63610c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.310644 891 tablet_service.cc:1468] Processing CreateTablet for tablet 1a42bf5f0ad84a43a8de1a7a39138d0e (DEFAULT_TABLE table=default.Table_5d22cae7274d4d64811664493a162782 [id=7cd4907ca19e486d9d2390bc84a0018a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.310999 884 tablet_service.cc:1468] Processing CreateTablet for tablet 776d546c733f4d1d8bbbdd64e23e9e62 (DEFAULT_TABLE table=default.Table_610e4cc494e244878a60d2382727f85c [id=c143510d38964470815db0340ebd2ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.311545 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_65d439aaf9c9480f8d1b9a3889e23a7e" } new_table_name: "default.Table_968458660b3b479aa6fc0afcbd1d5925" modify_external_catalogs: true
I20250625 13:52:10.312029 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet af934858a4af4621bb55eb5a528fa14c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.312723 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff804c2097c04317aa185379facaf553" } new_table_name: "default.Table_29be1cb29e48403cb7e0c22d2d3ab7bd" modify_external_catalogs: true
I20250625 13:52:10.313077 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "430c3bed360b4f32855a8adb99076608" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:10.313827 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_85855211d1e441d48d2aeb29e80d3dd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.314337 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85855211d1e441d48d2aeb29e80d3dd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.314821 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 9ec33dce871a442ea93577db109d42e6 (DEFAULT_TABLE table=default.Table_b45de936fca240bd8d8d63cac7007f5e [id=0af3ac93e56643548b3da06116a13b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.315182 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec33dce871a442ea93577db109d42e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.315387 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_464af049186b4c6d849a988e68947171"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.315735 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:10.315819 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_464af049186b4c6d849a988e68947171 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.317201 1014 tablet_service.cc:1468] Processing CreateTablet for tablet ea383a1ceddc461fb2531dbca714029e (DEFAULT_TABLE table=default.Table_d47cfb3324a443628070fd4437024a71 [id=cc661f7c95d442f881c5ce0ef5eec921]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.317556 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea383a1ceddc461fb2531dbca714029e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.318054 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 1068b65734bc4c29ba8b09a7a3d575b1 (DEFAULT_TABLE table=default.Table_338af88ef39143f0823658cf2b61c83b [id=c7b896b672834d3f8d4e698685034436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.318380 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1068b65734bc4c29ba8b09a7a3d575b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.319027 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 776d546c733f4d1d8bbbdd64e23e9e62 (DEFAULT_TABLE table=default.Table_610e4cc494e244878a60d2382727f85c [id=c143510d38964470815db0340ebd2ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.319296 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776d546c733f4d1d8bbbdd64e23e9e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.319356 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_628c945c37c84b88a1ed70deaadd49a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.319837 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_628c945c37c84b88a1ed70deaadd49a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.320914 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_92b3804731fc429a8b6412d0be16b8ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.321393 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92b3804731fc429a8b6412d0be16b8ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.323894 597 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
W20250625 13:52:10.324414 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f4d6be016284dc796945270178b9ee1 consensus state has no leader
W20250625 13:52:10.325460 711 leader_election.cc:343] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.325640 711 leader_election.cc:304] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:10.326143 2213 raft_consensus.cc:2747] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:10.335800 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f44d0f54bc24feb8386cfce1cd67db6 consensus state has no leader
I20250625 13:52:10.336591 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.337025 1165 ts_tablet_manager.cc:1918] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:10.337147 1165 log.cc:1199] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/6c5fc63828d9408b9cf774a966f43cf2
I20250625 13:52:10.337481 1165 ts_tablet_manager.cc:1939] T 6c5fc63828d9408b9cf774a966f43cf2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:52:10.339582 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e2711256c9042ebb651ae0ca3d02c89 consensus state has no leader
I20250625 13:52:10.340945 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e7c95bd30e403f9cd391f01ccec124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.342650 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 6c5fc63828d9408b9cf774a966f43cf2 (table default.Table_de6ae2243e604e0a8094e43c5b76ca27 [id=6de15b35e99646eb8eba9c11eea579c2]) successfully deleted
W20250625 13:52:10.342949 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a89f9ffb5e4643f8915e9fd69dd3e842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a89f9ffb5e4643f8915e9fd69dd3e842 already in progress: creating tablet
I20250625 13:52:10.343564 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ccd2404d954c6393476fdbcdc5d37a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.343761 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 156da2a965c64d1090a4f9d4eafdec70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156da2a965c64d1090a4f9d4eafdec70 already in progress: creating tablet
W20250625 13:52:10.344007 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0012e95ff4924723a6e6aaf6b85da438 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0012e95ff4924723a6e6aaf6b85da438 already in progress: creating tablet
I20250625 13:52:10.344079 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec33dce871a442ea93577db109d42e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.344446 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet af934858a4af4621bb55eb5a528fa14c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af934858a4af4621bb55eb5a528fa14c already in progress: creating tablet
W20250625 13:52:10.344664 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 already in progress: creating tablet
W20250625 13:52:10.345185 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet abc44a41b30e4a53977ec81d6905612f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc44a41b30e4a53977ec81d6905612f already in progress: creating tablet
W20250625 13:52:10.345381 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d5cb3b35a5c2483fa82226133a7c161d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5cb3b35a5c2483fa82226133a7c161d already in progress: creating tablet
I20250625 13:52:10.345999 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2e590dac73914afca2846962795030aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.347090 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e590dac73914afca2846962795030aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.348001 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a3634181d607460c9e84b9cca878d57b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.348497 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3634181d607460c9e84b9cca878d57b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.350221 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6895050a06634c5797ba2c2af0f4da14" } new_table_name: "default.Table_7a541fb5125f4b29bff6063faf68f7e7" modify_external_catalogs: true
W20250625 13:52:10.353561 849 leader_election.cc:343] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.353766 849 leader_election.cc:304] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:10.354251 849 leader_election.cc:343] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:10.354390 849 leader_election.cc:304] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:10.354645 2204 raft_consensus.cc:2747] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:10.354825 2204 raft_consensus.cc:2747] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:10.357004 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 1a42bf5f0ad84a43a8de1a7a39138d0e (DEFAULT_TABLE table=default.Table_5d22cae7274d4d64811664493a162782 [id=7cd4907ca19e486d9d2390bc84a0018a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.357390 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a42bf5f0ad84a43a8de1a7a39138d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.357600 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 21ccd2404d954c6393476fdbcdc5d37a (DEFAULT_TABLE table=default.Table_458d788e23274f5d9f4044fe32959484 [id=c6ec184916a8470eaf500367cfe4e56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.357779 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 39e7c95bd30e403f9cd391f01ccec124 (DEFAULT_TABLE table=default.Table_47e85ba24d824e5eaf587f4dbc7c9d22 [id=3ee1e37050ad4e958797be261e5dc440]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.357862 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776d546c733f4d1d8bbbdd64e23e9e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.357903 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ccd2404d954c6393476fdbcdc5d37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.358028 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e7c95bd30e403f9cd391f01ccec124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.358634 1012 tablet_service.cc:1468] Processing CreateTablet for tablet ed3f5984ecde4a81aaba0802b69a5ccc (DEFAULT_TABLE table=default.Table_3cac432f7e1f4bf28577a482f3cab0df [id=f8aba8cd9bdd478f9feb9ae521795239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.358914 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed3f5984ecde4a81aaba0802b69a5ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.359465 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 146ca3d1da24429e8487886e5e5efd6b (DEFAULT_TABLE table=default.Table_c9b44fa6c36f4b0fab24634ea2235568 [id=06e1a3dddf984b508ec13579bd9322b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.361455 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a42bf5f0ad84a43a8de1a7a39138d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.361649 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1068b65734bc4c29ba8b09a7a3d575b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.365306 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 781) took 2422 ms. Trace:
I20250625 13:52:10.365407 1168 rpcz_store.cc:276] 0625 13:52:07.942481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.942668 (+ 187us) service_pool.cc:225] Handling call
0625 13:52:09.879207 (+1936539us) ts_tablet_manager.cc:1145] Deleting tablet 36b151e200e1430c99ac194bc55881c1
0625 13:52:09.879217 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:09.985415 (+106198us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:10.081170 (+ 95755us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:10.365250 (+284080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1902798}
I20250625 13:52:10.365948 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 782) took 2417 ms. Trace:
I20250625 13:52:10.366019 1168 rpcz_store.cc:276] 0625 13:52:07.948779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.948836 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:10.365824 (+2416988us) ts_tablet_manager.cc:1145] Deleting tablet 171ce24a435e40e48026039941ca7883
0625 13:52:10.365833 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.365933 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2389373}
I20250625 13:52:10.366426 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 783) took 2406 ms. Trace:
I20250625 13:52:10.366505 1168 rpcz_store.cc:276] 0625 13:52:07.959837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.960150 (+ 313us) service_pool.cc:225] Handling call
0625 13:52:10.366327 (+2406177us) ts_tablet_manager.cc:1145] Deleting tablet 47eb205a9bcd4608930d9355978a44ce
0625 13:52:10.366336 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.366414 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2387171}
I20250625 13:52:10.366905 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 784) took 2403 ms. Trace:
I20250625 13:52:10.366981 1168 rpcz_store.cc:276] 0625 13:52:07.963932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.963995 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:10.366790 (+2402795us) ts_tablet_manager.cc:1145] Deleting tablet 664a83b20b344584b5c85db53bfa9021
0625 13:52:10.366799 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.366890 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2387577}
I20250625 13:52:10.367326 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea383a1ceddc461fb2531dbca714029e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.367540 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 785) took 2395 ms. Trace:
I20250625 13:52:10.367605 1168 rpcz_store.cc:276] 0625 13:52:07.972370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.972426 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:10.367424 (+2394998us) ts_tablet_manager.cc:1145] Deleting tablet 944759869ced45f09dfad8959b32edff
0625 13:52:10.367432 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.367526 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2388168}
I20250625 13:52:10.368053 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 786) took 2386 ms. Trace:
I20250625 13:52:10.368116 1168 rpcz_store.cc:276] 0625 13:52:07.981543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:07.981594 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:10.367938 (+2386344us) ts_tablet_manager.cc:1145] Deleting tablet a2acecefed724eb3b55116b8e5452853
0625 13:52:10.367946 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.368039 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2386166}
I20250625 13:52:10.368502 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 787) took 2310 ms. Trace:
I20250625 13:52:10.368566 1168 rpcz_store.cc:276] 0625 13:52:08.058325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.058437 (+ 112us) service_pool.cc:225] Handling call
0625 13:52:10.368388 (+2309951us) ts_tablet_manager.cc:1145] Deleting tablet 490c8902efeb46c99543747a8787c1c7
0625 13:52:10.368395 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.368481 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2309753}
I20250625 13:52:10.371141 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
I20250625 13:52:10.371430 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed3f5984ecde4a81aaba0802b69a5ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.371409 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 723246e1e6824faf9f9d39c3ea941fe4 (table default.Table_49756985b3fc428cbc871659e5aabdc6 [id=1178d7027158455bb95716dd9e102ef9]). Expected version 2 got 0
I20250625 13:52:10.371614 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:10.371850 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:10.372009 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dc75a0a8421b457089cc707d9eb6b76f (table default.Table_f618b45e019044ea946538ed3fa05471 [id=fa0246aff3f648538e6fddcb30f3a399]). Expected version 3 got 0
I20250625 13:52:10.373940 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 790) took 2255 ms. Trace:
I20250625 13:52:10.374009 1168 rpcz_store.cc:276] 0625 13:52:08.115780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.115832 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:10.370396 (+2254564us) ts_tablet_manager.cc:1145] Deleting tablet 455884b62ee44b6da53543093f721027
0625 13:52:10.370408 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.371146 (+ 738us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254076}
I20250625 13:52:10.377390 2210 raft_consensus.cc:491] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.377553 2210 raft_consensus.cc:513] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.378343 2210 leader_election.cc:290] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:10.379854 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_002bf8bcbdb849c2988585953c8828b0" } modify_external_catalogs: true
I20250625 13:52:10.380635 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_002bf8bcbdb849c2988585953c8828b0" } modify_external_catalogs: true
I20250625 13:52:10.382207 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2bcf5e3903254063bd1d49f7f9455737" } modify_external_catalogs: true
I20250625 13:52:10.382371 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2bcf5e3903254063bd1d49f7f9455737" } modify_external_catalogs: true
I20250625 13:52:10.359843 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 146ca3d1da24429e8487886e5e5efd6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.360540 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet d5cb3b35a5c2483fa82226133a7c161d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:10.389094 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f3de20ac5f945bdb62535258880186f consensus state has no leader
I20250625 13:52:10.395051 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561] alter complete (version 2)
I20250625 13:52:10.395860 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 8285f51c022548fa9c73c8dd84b2b8c1 (DEFAULT_TABLE table=default.Table_1e884456a1d24e429f5e791a07f1014f [id=9361b4021f3a4024b7608b0a6996ae44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.396226 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8285f51c022548fa9c73c8dd84b2b8c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.396824 2218 log.cc:927] Time spent T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20250625 13:52:10.399806 746 tablet_service.cc:1468] Processing CreateTablet for tablet 9265e1a660a84ef19863ccdbad72033d (DEFAULT_TABLE table=default.Table_fed63050407b40569f0a8911e9d1fd21 [id=6e495c98f9344b37afc59f947f596b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.403885 890 tablet_service.cc:1468] Processing CreateTablet for tablet 934976b12e674cce8d1e4f8fad60fc18 (DEFAULT_TABLE table=default.Table_717dce88b94246cab98733e4565a27b5 [id=214da52a59a148e3bee6ee3a40b85bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.404299 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934976b12e674cce8d1e4f8fad60fc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.404300 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 973c483c87b84ea09b6c7d556e54b490
I20250625 13:52:10.405552 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a1a9a69d26c14a62b26e0e90157782af
I20250625 13:52:10.409107 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bd3d18582db04d18829da98f82d6ee0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.409622 2201 leader_election.cc:290] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:10.409678 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd3d18582db04d18829da98f82d6ee0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.410996 895 tablet_service.cc:1468] Processing CreateTablet for tablet c930ac853a304fbab4be6fb37df5f79a (DEFAULT_TABLE table=default.Table_2a88884ec2db4ae8bc4e94280d4c5022 [id=f8f92568dbf84291896ba4e79bbb6614]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.411041 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_630c47cb56984efbac74836e412efaa6" } modify_external_catalogs: true
I20250625 13:52:10.411262 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_630c47cb56984efbac74836e412efaa6" } modify_external_catalogs: true
I20250625 13:52:10.411386 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c930ac853a304fbab4be6fb37df5f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.412336 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be3313ec5dfb41b68662d3ff4c8c94f9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:10.412495 912 raft_consensus.cc:2466] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:10.412860 983 leader_election.cc:304] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:10.413338 2198 consensus_queue.cc:1035] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:10.414098 2198 raft_consensus.cc:2802] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:10.414178 2198 raft_consensus.cc:491] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:10.414244 2198 raft_consensus.cc:3058] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:10.416213 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_88d5d2614816414f8a45b3526e1608d8" } modify_external_catalogs: true
I20250625 13:52:10.416563 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_88d5d2614816414f8a45b3526e1608d8" } modify_external_catalogs: true
I20250625 13:52:10.417243 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e03f752b3f2463b84f66809236ba398" } modify_external_catalogs: true
I20250625 13:52:10.417451 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e03f752b3f2463b84f66809236ba398" } modify_external_catalogs: true
I20250625 13:52:10.417908 877 tablet_service.cc:1468] Processing CreateTablet for tablet 7507d50e9f464043b8901491c57391d6 (DEFAULT_TABLE table=default.Table_893c6cc9fca542d0ab7c03d3f08207cc [id=6ddaa553802c431484cdd55c6615dcb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.418255 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7507d50e9f464043b8901491c57391d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.418399 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4d2946a0f06e4f3fa1cde181e94856f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.419020 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d2946a0f06e4f3fa1cde181e94856f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.419476 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b14d00c93a964edd815122e3723f89e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:10.421949 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 934976b12e674cce8d1e4f8fad60fc18 (DEFAULT_TABLE table=default.Table_717dce88b94246cab98733e4565a27b5 [id=214da52a59a148e3bee6ee3a40b85bb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.422281 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 934976b12e674cce8d1e4f8fad60fc18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.422673 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5b350adad81545bc9d5ee61caadc1f5c (DEFAULT_TABLE table=default.Table_31d921b4f7ca4806bf89fd8bca5e4e4c [id=d2d32cb58b334706869376417ab2c230]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.424597 892 tablet_service.cc:1468] Processing CreateTablet for tablet 47c70919181d4c5392f65df221ec6348 (DEFAULT_TABLE table=default.Table_6e03f752b3f2463b84f66809236ba398 [id=40cad82310ae482fa8d283f5cc888d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.424768 893 tablet_service.cc:1468] Processing CreateTablet for tablet b43341acdd754726ac0fbb06d27eec1c (DEFAULT_TABLE table=default.Table_e3936edb9d8e47d08189e8903287c572 [id=06c61eaf6373492e8a72ba7d0442c601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.425019 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c70919181d4c5392f65df221ec6348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.425240 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43341acdd754726ac0fbb06d27eec1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.425808 1018 tablet_service.cc:1468] Processing CreateTablet for tablet b43341acdd754726ac0fbb06d27eec1c (DEFAULT_TABLE table=default.Table_e3936edb9d8e47d08189e8903287c572 [id=06c61eaf6373492e8a72ba7d0442c601]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.426124 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43341acdd754726ac0fbb06d27eec1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.429628 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_15e2aec48b1c4191a1ca7a3b48eac9ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.430102 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15e2aec48b1c4191a1ca7a3b48eac9ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.430513 1028 tablet_service.cc:1468] Processing CreateTablet for tablet c930ac853a304fbab4be6fb37df5f79a (DEFAULT_TABLE table=default.Table_2a88884ec2db4ae8bc4e94280d4c5022 [id=f8f92568dbf84291896ba4e79bbb6614]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.430855 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c930ac853a304fbab4be6fb37df5f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.431339 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9265e1a660a84ef19863ccdbad72033d (DEFAULT_TABLE table=default.Table_fed63050407b40569f0a8911e9d1fd21 [id=6e495c98f9344b37afc59f947f596b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.431702 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9265e1a660a84ef19863ccdbad72033d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.431638 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7d1b445e65344381aac8680dad76a4b2" } new_table_name: "default.Table_aca9eb55cf01477092c2c2288bdab60b" modify_external_catalogs: true
I20250625 13:52:10.437300 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7507d50e9f464043b8901491c57391d6 (DEFAULT_TABLE table=default.Table_893c6cc9fca542d0ab7c03d3f08207cc [id=6ddaa553802c431484cdd55c6615dcb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.437646 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7507d50e9f464043b8901491c57391d6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.438735 1821 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.004s sys 0.008s
W20250625 13:52:10.439218 1821 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.004s sys 0.008s
I20250625 13:52:10.439340 1821 raft_consensus.cc:695] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:10.439654 1821 consensus_queue.cc:237] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.441624 2213 raft_consensus.cc:491] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.441799 2213 raft_consensus.cc:513] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:10.441883 849 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.62:33015, user_credentials={real_user=slave}} blocked reactor thread for 59727.8us
I20250625 13:52:10.443792 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_f618b45e019044ea946538ed3fa05471 [id=fa0246aff3f648538e6fddcb30f3a399] alter complete (version 3)
W20250625 13:52:10.444366 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd7a6587aa674a0eaaa107e2a2f673a3 consensus state has no leader
W20250625 13:52:10.444396 849 leader_election.cc:343] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.444780 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e3939ef22ea344328721dfba67d933d1
I20250625 13:52:10.450215 747 tablet_service.cc:1468] Processing CreateTablet for tablet 170ffbff76af447a83d663fa8688b7a2 (DEFAULT_TABLE table=default.Table_3774c3c6876d4c799d85694da32c4f74 [id=afebefb5201a4a05926a98a5044c2b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.450773 758 tablet_service.cc:1468] Processing CreateTablet for tablet 8f88091205b944b386094e50cf92c36b (DEFAULT_TABLE table=default.Table_66dd8f78c67643d9828dee272954c648 [id=de1e2840226b4163b1092cae2e0d90c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.451735 753 tablet_service.cc:1468] Processing CreateTablet for tablet c00d7323dcae453683206e12ad22e1c8 (DEFAULT_TABLE table=default.Table_85881e844bd44e0e97ff03992f7359c1 [id=ce0e626eb1e34c64a3423df6bcb6891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.453655 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 74884bf313724d678d3b5f5869f09f97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.454627 1165 tablet_replica.cc:331] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:10.454864 1165 raft_consensus.cc:2241] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:10.455056 1165 raft_consensus.cc:2270] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:10.457931 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 9265e1a660a84ef19863ccdbad72033d (DEFAULT_TABLE table=default.Table_fed63050407b40569f0a8911e9d1fd21 [id=6e495c98f9344b37afc59f947f596b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.458353 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9265e1a660a84ef19863ccdbad72033d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.459728 878 tablet_service.cc:1468] Processing CreateTablet for tablet 3562e8b8731a4d76bb1a36189f957e07 (DEFAULT_TABLE table=default.Table_d8054fa43c994aab98b1d5541d0be6f9 [id=82ac4f198e8242fc9a95787dc3331c8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.460101 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3562e8b8731a4d76bb1a36189f957e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.467978 2213 leader_election.cc:290] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:10.468353 754 tablet_service.cc:1468] Processing CreateTablet for tablet 413502c39dd04c02b088071a445b69c3 (DEFAULT_TABLE table=default.Table_7389669b9b714262b43171af80118da6 [id=0b6a5e62c8474868ba28d6229d947516]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.468647 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:10.469676 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74884bf313724d678d3b5f5869f09f97" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:10.470229 711 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.470873 1165 ts_tablet_manager.cc:1905] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:10.470888 742 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1e660c5b244b1c954a1c6426fd959e (DEFAULT_TABLE table=default.Table_05a86981b085438083683deb71ee651d [id=4394b6fcfdf943a9928991a8f2825126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.470922 2198 raft_consensus.cc:513] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.472290 2198 leader_election.cc:290] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:10.476025 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:10.480746 749 tablet_service.cc:1468] Processing CreateTablet for tablet aafe89794fce4a56b50c4a79ded259a7 (DEFAULT_TABLE table=default.Table_0f4d14b4d5d84cab89f1ce9fa6a60e62 [id=86ed36d73e5040c4bcec5679c9b543d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.482934 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9265e1a660a84ef19863ccdbad72033d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.483120 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 170ffbff76af447a83d663fa8688b7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.483440 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413502c39dd04c02b088071a445b69c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.483685 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be3313ec5dfb41b68662d3ff4c8c94f9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:10.483880 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00d7323dcae453683206e12ad22e1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.483919 912 raft_consensus.cc:3058] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:10.484061 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b350adad81545bc9d5ee61caadc1f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.484161 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafe89794fce4a56b50c4a79ded259a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.487130 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f88091205b944b386094e50cf92c36b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.488502 851 leader_election.cc:343] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b14d00c93a964edd815122e3723f89e7
I20250625 13:52:10.488634 851 leader_election.cc:304] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:10.489979 751 tablet_service.cc:1468] Processing CreateTablet for tablet c08720d4fafa4784ab5dbab02c725545 (DEFAULT_TABLE table=default.Table_94eb551cf0ef413b871b98aff36fcdf0 [id=9249a07a87fe4c7fbd26e07b9b968082]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.490334 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08720d4fafa4784ab5dbab02c725545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.491933 755 tablet_service.cc:1468] Processing CreateTablet for tablet d60f5d5746d44a8a8bb4a317d15f306b (DEFAULT_TABLE table=default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.492264 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60f5d5746d44a8a8bb4a317d15f306b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.492734 912 raft_consensus.cc:2466] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:10.493196 983 leader_election.cc:304] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:10.493495 2198 raft_consensus.cc:2802] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:10.493588 2198 raft_consensus.cc:695] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:10.493804 2198 consensus_queue.cc:237] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.494508 740 tablet_service.cc:1468] Processing CreateTablet for tablet 63f82f4a24d44888b222c93f5d614ba5 (DEFAULT_TABLE table=default.Table_ea94ac9a52d84ddda29e98fa67d38868 [id=f8d584b01250470c9883d9ee8596d519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.494848 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63f82f4a24d44888b222c93f5d614ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.494977 2210 raft_consensus.cc:2747] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:10.497574 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8d39a96a128943438ead46a70f47c798 (DEFAULT_TABLE table=default.Table_c9744a7c82cf465696fd5127da2131ba [id=862bc721f3cf4ba794a56b6ea8ce53a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.497916 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d39a96a128943438ead46a70f47c798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.501411 2162 log.cc:826] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Log is configured to *not* fsync() on all Append() calls
I20250625 13:52:10.502091 1165 ts_tablet_manager.cc:1918] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:10.502197 1165 log.cc:1199] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/74884bf313724d678d3b5f5869f09f97
I20250625 13:52:10.505769 1165 ts_tablet_manager.cc:1939] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:10.507243 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 74884bf313724d678d3b5f5869f09f97 (table default.Table_25d9c73fdf1b43b787345769f54915d6 [id=79ace8a081c447a5b4c44b5911a906c8]) successfully deleted
I20250625 13:52:10.508075 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1e660c5b244b1c954a1c6426fd959e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.509260 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 47c70919181d4c5392f65df221ec6348
I20250625 13:52:10.513651 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bd4cc91d274a45a393beffaaa59a35f0
I20250625 13:52:10.520978 1171 ts_tablet_manager.cc:1428] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.436s user 0.019s sys 0.003s
I20250625 13:52:10.519232 745 tablet_service.cc:1468] Processing CreateTablet for tablet 29758dd57b8d49e8869510c41122b97e (DEFAULT_TABLE table=default.Table_1ec436c40a764dbba4243d2522dcfb1e [id=5758233a9d5444f3be39936851eca9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.524008 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 973c483c87b84ea09b6c7d556e54b490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.524224 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a1a9a69d26c14a62b26e0e90157782af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.525409 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29758dd57b8d49e8869510c41122b97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.527335 1171 tablet_bootstrap.cc:492] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:10.527475 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5bc830734d3046dcafc7d1ccc0c5c561" } new_table_name: "default.Table_ba8009758e5749ca8afc871aa72f9430" modify_external_catalogs: true
I20250625 13:52:10.529001 1171 tablet_bootstrap.cc:654] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:10.530231 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e3939ef22ea344328721dfba67d933d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.530592 750 tablet_service.cc:1468] Processing CreateTablet for tablet d12f07f4145240958d7decf21c03b6bc (DEFAULT_TABLE table=default.Table_464af049186b4c6d849a988e68947171 [id=bd1cd6099a8244348047ebb27abf3590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.530944 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12f07f4145240958d7decf21c03b6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.535703 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 47c70919181d4c5392f65df221ec6348 (DEFAULT_TABLE table=default.Table_6e03f752b3f2463b84f66809236ba398 [id=40cad82310ae482fa8d283f5cc888d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.536096 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c70919181d4c5392f65df221ec6348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.540032 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2bea8ee8d6074d3785dc33d166fc3485" } new_table_name: "default.Table_b8283f07d5f4442bbbf72894e932bd3a" modify_external_catalogs: true
I20250625 13:52:10.540127 885 tablet_service.cc:1468] Processing CreateTablet for tablet d0174345facc4216b6ddeb8f8c73790d (DEFAULT_TABLE table=default.Table_4d11b567f2c14b62a446981c6ad894d4 [id=665f17d9e587412197dc3304230fcc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.540535 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0174345facc4216b6ddeb8f8c73790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.543565 894 tablet_service.cc:1468] Processing CreateTablet for tablet d151560702a547acb01a353cf1fb27e3 (DEFAULT_TABLE table=default.Table_107ccce5d138433db4fb8351f69e40a0 [id=423c4d58d233464ebe4fcd7a7d426f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.543903 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d151560702a547acb01a353cf1fb27e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.544281 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf962c47820f45e79e0e70f9208a17ff consensus state has no leader
I20250625 13:52:10.546751 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ec4be031b7ee44a59b7015fda0f6bf44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.547396 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec4be031b7ee44a59b7015fda0f6bf44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.547796 887 tablet_service.cc:1468] Processing CreateTablet for tablet a1263945be894618a7f9946e7ca50efd (DEFAULT_TABLE table=default.Table_ac6acb4e2c0e4001927ffb4230ad2ecf [id=bc1d4ec693154bb5aa80709fed7d41e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.548112 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1263945be894618a7f9946e7ca50efd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.548755 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4181bbb4af9c4141b587968cb7ef8251 consensus state has no leader
I20250625 13:52:10.546753 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_974313f23d974d75914dc9b2f7bd25d9" } new_table_name: "default.Table_5e23db69da0c4ecd974b3ce372f254b7" modify_external_catalogs: true
I20250625 13:52:10.551612 883 tablet_service.cc:1468] Processing CreateTablet for tablet 413502c39dd04c02b088071a445b69c3 (DEFAULT_TABLE table=default.Table_7389669b9b714262b43171af80118da6 [id=0b6a5e62c8474868ba28d6229d947516]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.552032 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413502c39dd04c02b088071a445b69c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.555586 896 tablet_service.cc:1468] Processing CreateTablet for tablet 5b350adad81545bc9d5ee61caadc1f5c (DEFAULT_TABLE table=default.Table_31d921b4f7ca4806bf89fd8bca5e4e4c [id=d2d32cb58b334706869376417ab2c230]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.555941 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b350adad81545bc9d5ee61caadc1f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.556782 891 tablet_service.cc:1468] Processing CreateTablet for tablet 170ffbff76af447a83d663fa8688b7a2 (DEFAULT_TABLE table=default.Table_3774c3c6876d4c799d85694da32c4f74 [id=afebefb5201a4a05926a98a5044c2b9f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:10.557044 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2e6fbf3d81547f798b246a8b8f61387 consensus state has no leader
I20250625 13:52:10.557118 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 170ffbff76af447a83d663fa8688b7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.559535 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8f88091205b944b386094e50cf92c36b (DEFAULT_TABLE table=default.Table_66dd8f78c67643d9828dee272954c648 [id=de1e2840226b4163b1092cae2e0d90c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.559855 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f88091205b944b386094e50cf92c36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.560914 882 tablet_service.cc:1468] Processing CreateTablet for tablet c00d7323dcae453683206e12ad22e1c8 (DEFAULT_TABLE table=default.Table_85881e844bd44e0e97ff03992f7359c1 [id=ce0e626eb1e34c64a3423df6bcb6891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.561213 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00d7323dcae453683206e12ad22e1c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.561203 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet fd7a6587aa674a0eaaa107e2a2f673a3 (table default.Table_aca9eb55cf01477092c2c2288bdab60b [id=f63d1eacefa9422a97784e55c027ac35], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd7a6587aa674a0eaaa107e2a2f673a3 consensus state has no leader
I20250625 13:52:10.563781 891 tablet_service.cc:1468] Processing CreateTablet for tablet c08720d4fafa4784ab5dbab02c725545 (DEFAULT_TABLE table=default.Table_94eb551cf0ef413b871b98aff36fcdf0 [id=9249a07a87fe4c7fbd26e07b9b968082]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.563832 879 tablet_service.cc:1468] Processing CreateTablet for tablet d60f5d5746d44a8a8bb4a317d15f306b (DEFAULT_TABLE table=default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.564172 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08720d4fafa4784ab5dbab02c725545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.564163 881 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1e660c5b244b1c954a1c6426fd959e (DEFAULT_TABLE table=default.Table_05a86981b085438083683deb71ee651d [id=4394b6fcfdf943a9928991a8f2825126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.564514 880 tablet_service.cc:1468] Processing CreateTablet for tablet aafe89794fce4a56b50c4a79ded259a7 (DEFAULT_TABLE table=default.Table_0f4d14b4d5d84cab89f1ce9fa6a60e62 [id=86ed36d73e5040c4bcec5679c9b543d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.569052 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 36b151e200e1430c99ac194bc55881c1 (table default.Table_29924a11586f4d2bb3d0f8574a7849f8 [id=e057b2ec6f914a01b474481bc5c25349]) successfully deleted
W20250625 13:52:10.569375 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 171ce24a435e40e48026039941ca7883 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 171ce24a435e40e48026039941ca7883 already in progress: creating tablet
W20250625 13:52:10.569569 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 47eb205a9bcd4608930d9355978a44ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47eb205a9bcd4608930d9355978a44ce already in progress: creating tablet
W20250625 13:52:10.569816 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 664a83b20b344584b5c85db53bfa9021 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 664a83b20b344584b5c85db53bfa9021 already in progress: creating tablet
W20250625 13:52:10.570019 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 944759869ced45f09dfad8959b32edff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 944759869ced45f09dfad8959b32edff already in progress: creating tablet
W20250625 13:52:10.570231 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a2acecefed724eb3b55116b8e5452853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2acecefed724eb3b55116b8e5452853 already in progress: creating tablet
I20250625 13:52:10.570212 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3562e8b8731a4d76bb1a36189f957e07 (DEFAULT_TABLE table=default.Table_d8054fa43c994aab98b1d5541d0be6f9 [id=82ac4f198e8242fc9a95787dc3331c8c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:10.570428 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 490c8902efeb46c99543747a8787c1c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 490c8902efeb46c99543747a8787c1c7 already in progress: creating tablet
I20250625 13:52:10.570649 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3562e8b8731a4d76bb1a36189f957e07. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.570633 581 catalog_manager.cc:4895] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 455884b62ee44b6da53543093f721027 because the tablet was not found. No further retry: Not found: Tablet not found: 455884b62ee44b6da53543093f721027
I20250625 13:52:10.571642 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1e660c5b244b1c954a1c6426fd959e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.575645 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafe89794fce4a56b50c4a79ded259a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.579540 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1c72b998c9314821835754a40d9c88a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.580221 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c72b998c9314821835754a40d9c88a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.580623 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e40aed94ebc340d8b6324a1bd2840e03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.581085 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e40aed94ebc340d8b6324a1bd2840e03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.581843 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_513b6609ffeb48e592b807215f4e9488"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.582278 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_513b6609ffeb48e592b807215f4e9488 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.582571 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60f5d5746d44a8a8bb4a317d15f306b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.583389 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fd92384a196c4ca1a02f43763f639a77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.583853 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd92384a196c4ca1a02f43763f639a77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.592536 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_639d5431f288425b907989942e4aa7f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.593014 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_639d5431f288425b907989942e4aa7f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.594058 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aed3405f8cc04d6788a5b5ba519f5363" } new_table_name: "default.Table_81ed9530630f4ac9b888925f6f187d04" modify_external_catalogs: true
I20250625 13:52:10.594607 891 tablet_service.cc:1468] Processing CreateTablet for tablet 63f82f4a24d44888b222c93f5d614ba5 (DEFAULT_TABLE table=default.Table_ea94ac9a52d84ddda29e98fa67d38868 [id=f8d584b01250470c9883d9ee8596d519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.598553 884 tablet_service.cc:1468] Processing CreateTablet for tablet 8d39a96a128943438ead46a70f47c798 (DEFAULT_TABLE table=default.Table_c9744a7c82cf465696fd5127da2131ba [id=862bc721f3cf4ba794a56b6ea8ce53a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.599017 888 tablet_service.cc:1468] Processing CreateTablet for tablet 29758dd57b8d49e8869510c41122b97e (DEFAULT_TABLE table=default.Table_1ec436c40a764dbba4243d2522dcfb1e [id=5758233a9d5444f3be39936851eca9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.599105 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d39a96a128943438ead46a70f47c798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.599336 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29758dd57b8d49e8869510c41122b97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.603121 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a2251328ba8745ae98b9fead6e8e516e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.603775 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2251328ba8745ae98b9fead6e8e516e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.605629 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 973c483c87b84ea09b6c7d556e54b490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.605870 890 tablet_service.cc:1515] Processing DeleteTablet for tablet a1a9a69d26c14a62b26e0e90157782af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.606035 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e3939ef22ea344328721dfba67d933d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.606415 890 tablet_service.cc:1468] Processing CreateTablet for tablet d12f07f4145240958d7decf21c03b6bc (DEFAULT_TABLE table=default.Table_464af049186b4c6d849a988e68947171 [id=bd1cd6099a8244348047ebb27abf3590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.606906 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12f07f4145240958d7decf21c03b6bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.607730 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c70919181d4c5392f65df221ec6348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.607908 895 tablet_service.cc:1515] Processing DeleteTablet for tablet bd4cc91d274a45a393beffaaa59a35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.608055 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.608382 895 tablet_service.cc:1468] Processing CreateTablet for tablet fbc14c29aed345c9a26c8964ff094da4 (DEFAULT_TABLE table=default.Table_85855211d1e441d48d2aeb29e80d3dd5 [id=7863d92166ff4ccab85ed4ad79bea409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.608955 893 tablet_service.cc:1468] Processing CreateTablet for tablet 2593db823e974f1eb653470d5a23338f (DEFAULT_TABLE table=default.Table_628c945c37c84b88a1ed70deaadd49a6 [id=b4206cbbd39643549beb76e6025e6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.608955 877 tablet_service.cc:1468] Processing CreateTablet for tablet 754c690dd747460eafbd13ea9efdc692 (DEFAULT_TABLE table=default.Table_92b3804731fc429a8b6412d0be16b8ad [id=43af99dd61534e2aaba6c9f442ca34a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.609787 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7f21a5a5b40740a28dd60ad19524d6e4" } modify_external_catalogs: true
I20250625 13:52:10.609987 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7f21a5a5b40740a28dd60ad19524d6e4" } modify_external_catalogs: true
I20250625 13:52:10.611415 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20e9351636684a39a1d996846e778bb4" } modify_external_catalogs: true
I20250625 13:52:10.611585 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_20e9351636684a39a1d996846e778bb4" } modify_external_catalogs: true
W20250625 13:52:10.613000 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9710609da88d4b42993b3aef02416543 consensus state has no leader
I20250625 13:52:10.619473 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 01b8bd1d9fb1470eb1bf425aa8c7af51
I20250625 13:52:10.620143 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2593db823e974f1eb653470d5a23338f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.620615 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63f82f4a24d44888b222c93f5d614ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.621731 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754c690dd747460eafbd13ea9efdc692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.623461 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbc14c29aed345c9a26c8964ff094da4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.632704 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet af934858a4af4621bb55eb5a528fa14c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af934858a4af4621bb55eb5a528fa14c already in progress: creating tablet
W20250625 13:52:10.633154 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 already in progress: creating tablet
W20250625 13:52:10.633430 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d5cb3b35a5c2483fa82226133a7c161d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5cb3b35a5c2483fa82226133a7c161d already in progress: creating tablet
I20250625 13:52:10.635332 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.643003 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_065463f5287f4182ace6007e8cfcec03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.643846 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_065463f5287f4182ace6007e8cfcec03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.648963 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2c513d91226c4bf6aa6643cc25131331" } modify_external_catalogs: true
I20250625 13:52:10.649508 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_87f21414bdc44bddaaa231cd2dfc7ecf" } new_table_name: "default.Table_86e757b5d1f74c6a8d94cae99ec7a974" modify_external_catalogs: true
I20250625 13:52:10.649451 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_36b5354e955b4364845c9fbb1e93d0eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.650108 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36b5354e955b4364845c9fbb1e93d0eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.650230 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3765a1c34b154d1f97898bad646c9ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.650743 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3765a1c34b154d1f97898bad646c9ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.660055 744 tablet_service.cc:1468] Processing CreateTablet for tablet b55fd2d3609844d5ba9777190bb6e0b4 (DEFAULT_TABLE table=default.Table_f4e7c3d0a0cb4e4a94d0a2e2e63610c4 [id=744f62b816aa44fbbb70c13f46818a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.660472 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b55fd2d3609844d5ba9777190bb6e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.608398 884 tablet_service.cc:1468] Processing CreateTablet for tablet b55fd2d3609844d5ba9777190bb6e0b4 (DEFAULT_TABLE table=default.Table_f4e7c3d0a0cb4e4a94d0a2e2e63610c4 [id=744f62b816aa44fbbb70c13f46818a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.661269 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b55fd2d3609844d5ba9777190bb6e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.664844 896 tablet_service.cc:1468] Processing CreateTablet for tablet 8684eb9104e44227944b21784c54e496 (DEFAULT_TABLE table=default.Table_4d2946a0f06e4f3fa1cde181e94856f3 [id=7bef6cfdaece44549646dc74b108bada]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.665199 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8684eb9104e44227944b21784c54e496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.665831 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c9ae350a9dbf476081dc56a709de42c8
I20250625 13:52:10.668176 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2c513d91226c4bf6aa6643cc25131331" } modify_external_catalogs: true
I20250625 13:52:10.671733 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c70919181d4c5392f65df221ec6348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.674671 754 tablet_service.cc:1468] Processing CreateTablet for tablet fbc14c29aed345c9a26c8964ff094da4 (DEFAULT_TABLE table=default.Table_85855211d1e441d48d2aeb29e80d3dd5 [id=7863d92166ff4ccab85ed4ad79bea409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.675136 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbc14c29aed345c9a26c8964ff094da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.678051 879 tablet_service.cc:1468] Processing CreateTablet for tablet d64d8f832da048e68e688a037b593719 (DEFAULT_TABLE table=default.Table_2e590dac73914afca2846962795030aa [id=ddf2721f40274855986d81f26d332f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.678404 894 tablet_service.cc:1468] Processing CreateTablet for tablet 4111b95923ed4d73b7eca60b964d9494 (DEFAULT_TABLE table=default.Table_bd3d18582db04d18829da98f82d6ee0c [id=0b427c87bb6446daa95efbaed62b067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.680457 893 tablet_service.cc:1468] Processing CreateTablet for tablet 0421a7491bdc4152bf6b1ea0a5b53c9e (DEFAULT_TABLE table=default.Table_15e2aec48b1c4191a1ca7a3b48eac9ab [id=0a0b8545bca14836a8c5f102f328f9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.680872 889 tablet_service.cc:1468] Processing CreateTablet for tablet 30b5b3444fd245bbb2ecd5889d956862 (DEFAULT_TABLE table=default.Table_a3634181d607460c9e84b9cca878d57b [id=f6a1fa8c3e804c6d996be293a1e207ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.682271 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0421a7491bdc4152bf6b1ea0a5b53c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.686674 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30b5b3444fd245bbb2ecd5889d956862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.687536 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64d8f832da048e68e688a037b593719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.693003 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4111b95923ed4d73b7eca60b964d9494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.695133 748 tablet_service.cc:1515] Processing DeleteTablet for tablet bd4cc91d274a45a393beffaaa59a35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.702773 748 tablet_service.cc:1468] Processing CreateTablet for tablet 754c690dd747460eafbd13ea9efdc692 (DEFAULT_TABLE table=default.Table_92b3804731fc429a8b6412d0be16b8ad [id=43af99dd61534e2aaba6c9f442ca34a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.702988 883 tablet_service.cc:1515] Processing DeleteTablet for tablet c9ae350a9dbf476081dc56a709de42c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.703042 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2770194711ec4a86bb1354517b095f60
I20250625 13:52:10.709883 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74884bf313724d678d3b5f5869f09f97" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:10.711519 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_893c6cc9fca542d0ab7c03d3f08207cc" } modify_external_catalogs: true
I20250625 13:52:10.711773 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_893c6cc9fca542d0ab7c03d3f08207cc" } modify_external_catalogs: true
I20250625 13:52:10.704268 2215 raft_consensus.cc:491] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.712615 2215 raft_consensus.cc:513] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.714184 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:10.715261 1012 tablet_service.cc:1468] Processing CreateTablet for tablet d151560702a547acb01a353cf1fb27e3 (DEFAULT_TABLE table=default.Table_107ccce5d138433db4fb8351f69e40a0 [id=423c4d58d233464ebe4fcd7a7d426f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.715672 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d151560702a547acb01a353cf1fb27e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.717598 1168 tablet_replica.cc:331] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:10.717835 1168 raft_consensus.cc:2241] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:10.718021 1168 raft_consensus.cc:2270] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:10.718570 1168 ts_tablet_manager.cc:1905] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:10.719115 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a44fab4e30564e9fbbc8d2b351437052 consensus state has no leader
I20250625 13:52:10.719990 2192 raft_consensus.cc:491] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.720108 2192 raft_consensus.cc:513] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.720742 2192 leader_election.cc:290] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:10.721822 1020 tablet_service.cc:1468] Processing CreateTablet for tablet d0174345facc4216b6ddeb8f8c73790d (DEFAULT_TABLE table=default.Table_4d11b567f2c14b62a446981c6ad894d4 [id=665f17d9e587412197dc3304230fcc0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.722085 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a1263945be894618a7f9946e7ca50efd (DEFAULT_TABLE table=default.Table_ac6acb4e2c0e4001927ffb4230ad2ecf [id=bc1d4ec693154bb5aa80709fed7d41e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.722213 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0174345facc4216b6ddeb8f8c73790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.722432 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_49756985b3fc428cbc871659e5aabdc6" } modify_external_catalogs: true
I20250625 13:52:10.722608 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_49756985b3fc428cbc871659e5aabdc6" } modify_external_catalogs: true
W20250625 13:52:10.724767 985 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.025s sys 0.069s
W20250625 13:52:10.724947 985 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.025s sys 0.069s
W20250625 13:52:10.725104 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 515323us
W20250625 13:52:10.725744 2234 log.cc:927] Time spent T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250625 13:52:10.726421 1171 tablet_bootstrap.cc:492] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:10.726557 1171 ts_tablet_manager.cc:1397] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.199s user 0.003s sys 0.000s
I20250625 13:52:10.726524 2231 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:10.727522 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 413502c39dd04c02b088071a445b69c3 (DEFAULT_TABLE table=default.Table_7389669b9b714262b43171af80118da6 [id=0b6a5e62c8474868ba28d6229d947516]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.727898 1171 raft_consensus.cc:357] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.728060 1171 raft_consensus.cc:383] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.728104 1171 raft_consensus.cc:738] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.728307 1171 consensus_queue.cc:260] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.728886 1171 ts_tablet_manager.cc:1428] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250625 13:52:10.729135 1171 tablet_bootstrap.cc:492] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:10.729496 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 2770194711ec4a86bb1354517b095f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:10.730675 1171 tablet_bootstrap.cc:654] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:10.733075 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754c690dd747460eafbd13ea9efdc692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.734935 756 tablet_service.cc:1468] Processing CreateTablet for tablet 2593db823e974f1eb653470d5a23338f (DEFAULT_TABLE table=default.Table_628c945c37c84b88a1ed70deaadd49a6 [id=b4206cbbd39643549beb76e6025e6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.735378 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.735721 741 tablet_service.cc:1468] Processing CreateTablet for tablet 8684eb9104e44227944b21784c54e496 (DEFAULT_TABLE table=default.Table_4d2946a0f06e4f3fa1cde181e94856f3 [id=7bef6cfdaece44549646dc74b108bada]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.736094 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8684eb9104e44227944b21784c54e496. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.738199 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 973c483c87b84ea09b6c7d556e54b490 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 973c483c87b84ea09b6c7d556e54b490 already in progress: creating tablet
W20250625 13:52:10.738708 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a1a9a69d26c14a62b26e0e90157782af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a9a69d26c14a62b26e0e90157782af already in progress: creating tablet
W20250625 13:52:10.738791 851 consensus_peers.cc:487] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b129117db064a15b27e82439637cf22. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:10.738997 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e3939ef22ea344328721dfba67d933d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3939ef22ea344328721dfba67d933d1 already in progress: creating tablet
W20250625 13:52:10.739214 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 47c70919181d4c5392f65df221ec6348 because the tablet was not found. No further retry: Not found: Tablet not found: 47c70919181d4c5392f65df221ec6348
W20250625 13:52:10.739403 985 leader_election.cc:343] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: be3313ec5dfb41b68662d3ff4c8c94f9
W20250625 13:52:10.739794 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bd4cc91d274a45a393beffaaa59a35f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4cc91d274a45a393beffaaa59a35f0 already in progress: creating tablet
W20250625 13:52:10.739866 985 leader_election.cc:343] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: be3313ec5dfb41b68662d3ff4c8c94f9
I20250625 13:52:10.741390 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1263945be894618a7f9946e7ca50efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.742934 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c5fc63828d9408b9cf774a966f43cf2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:10.745721 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 5b350adad81545bc9d5ee61caadc1f5c (DEFAULT_TABLE table=default.Table_31d921b4f7ca4806bf89fd8bca5e4e4c [id=d2d32cb58b334706869376417ab2c230]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.737417 752 tablet_service.cc:1468] Processing CreateTablet for tablet 4111b95923ed4d73b7eca60b964d9494 (DEFAULT_TABLE table=default.Table_bd3d18582db04d18829da98f82d6ee0c [id=0b427c87bb6446daa95efbaed62b067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.746260 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4111b95923ed4d73b7eca60b964d9494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.747406 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b003a7ace5e74fafb5a936359d3cf7b5" } new_table_name: "default.Table_e3a43a004fde420b9c388ab931d7e3eb" modify_external_catalogs: true
W20250625 13:52:10.750013 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 already in progress: creating tablet
I20250625 13:52:10.750697 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2593db823e974f1eb653470d5a23338f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.753211 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9267df231f354c578c38f452418e523c (table default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4]). Expected version 1 got 0
I20250625 13:52:10.753721 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:10.753989 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:10.754160 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_94aa82f5ff874e7b826fcb2abc61eeb7 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 3 got 0
W20250625 13:52:10.754654 1170 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.012s sys 0.012s
W20250625 13:52:10.754765 1170 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.589s user 0.012s sys 0.012s
I20250625 13:52:10.749543 742 tablet_service.cc:1468] Processing CreateTablet for tablet d64d8f832da048e68e688a037b593719 (DEFAULT_TABLE table=default.Table_2e590dac73914afca2846962795030aa [id=ddf2721f40274855986d81f26d332f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.755656 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64d8f832da048e68e688a037b593719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.756279 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c389c3b02ed34ae6bb96dc94cb8d7d98" } new_table_name: "default.Table_50fd1f26e7c343008610cf804b2a8d0a" modify_external_catalogs: true
I20250625 13:52:10.758201 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_876b0fe22f3147528bd9aedba430dc9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.758720 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_876b0fe22f3147528bd9aedba430dc9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.759591 2192 raft_consensus.cc:491] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.759727 2192 raft_consensus.cc:513] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.760303 2192 leader_election.cc:290] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:10.761808 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_47e85ba24d824e5eaf587f4dbc7c9d22" } new_table_name: "default.Table_7191422e30b643f9a6ed910dc4b614a3" modify_external_catalogs: true
W20250625 13:52:10.763521 1047 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.005s sys 0.022s
W20250625 13:52:10.763634 1047 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.629s user 0.005s sys 0.022s
W20250625 13:52:10.764093 2207 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.021s sys 0.011s
W20250625 13:52:10.766074 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ac647b1d704142b94da8d506319466 consensus state has no leader
W20250625 13:52:10.764181 2207 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.565s user 0.021s sys 0.011s
W20250625 13:52:10.769965 849 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.770323 1842 consensus_queue.cc:1035] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:52:10.764232 1045 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.010s sys 0.018s
W20250625 13:52:10.771910 1045 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.010s sys 0.018s
I20250625 13:52:10.774009 1166 tablet_bootstrap.cc:492] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:10.774147 1166 ts_tablet_manager.cc:1397] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.493s user 0.020s sys 0.029s
W20250625 13:52:10.764302 1046 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.003s sys 0.026s
W20250625 13:52:10.777230 1046 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.004s sys 0.027s
W20250625 13:52:10.777397 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39e7c95bd30e403f9cd391f01ccec124 consensus state has no leader
W20250625 13:52:10.778213 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e2d109e6fce407dad72a2469479b5ed consensus state has no leader
I20250625 13:52:10.779726 749 tablet_service.cc:1468] Processing CreateTablet for tablet 30b5b3444fd245bbb2ecd5889d956862 (DEFAULT_TABLE table=default.Table_a3634181d607460c9e84b9cca878d57b [id=f6a1fa8c3e804c6d996be293a1e207ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.782475 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30b5b3444fd245bbb2ecd5889d956862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.785068 2215 leader_election.cc:290] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:10.789090 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c00d7323dcae453683206e12ad22e1c8 (DEFAULT_TABLE table=default.Table_85881e844bd44e0e97ff03992f7359c1 [id=ce0e626eb1e34c64a3423df6bcb6891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.791256 1170 tablet_bootstrap.cc:492] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:10.791396 1170 ts_tablet_manager.cc:1397] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.658s user 0.015s sys 0.013s
I20250625 13:52:10.793385 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1e660c5b244b1c954a1c6426fd959e (DEFAULT_TABLE table=default.Table_05a86981b085438083683deb71ee651d [id=4394b6fcfdf943a9928991a8f2825126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.798601 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 8f88091205b944b386094e50cf92c36b (DEFAULT_TABLE table=default.Table_66dd8f78c67643d9828dee272954c648 [id=de1e2840226b4163b1092cae2e0d90c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.799093 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ed63056903cc48609558b9b8419c6c8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.801046 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed63056903cc48609558b9b8419c6c8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.802316 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6b8809b44d10450497c7a9f223dda4de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.802829 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b8809b44d10450497c7a9f223dda4de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.803344 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 170ffbff76af447a83d663fa8688b7a2 (DEFAULT_TABLE table=default.Table_3774c3c6876d4c799d85694da32c4f74 [id=afebefb5201a4a05926a98a5044c2b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.805892 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2404e712e18146b68757bd77362c1358" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:10.806864 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f526b40a96af4910ab95446d509bf95b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.807552 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_52de821c06b24eea87c7839e7417a783"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.808008 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52de821c06b24eea87c7839e7417a783 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.809276 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_220446a0ea6a427d9940bdb6fd32dad7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.809706 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_220446a0ea6a427d9940bdb6fd32dad7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:10.810747 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f526b40a96af4910ab95446d509bf95b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.811053 743 tablet_service.cc:1468] Processing CreateTablet for tablet 0421a7491bdc4152bf6b1ea0a5b53c9e (DEFAULT_TABLE table=default.Table_15e2aec48b1c4191a1ca7a3b48eac9ab [id=0a0b8545bca14836a8c5f102f328f9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.811480 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0421a7491bdc4152bf6b1ea0a5b53c9e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.814783 849 leader_election.cc:343] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.819790 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6d08523c84e84dd487355d82976fc3c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.820354 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d08523c84e84dd487355d82976fc3c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.820477 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_464af049186b4c6d849a988e68947171" } new_table_name: "default.Table_06c39e7e718042edb9d263fd11f714f0" modify_external_catalogs: true
I20250625 13:52:10.822906 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4dfad83f70604649b15f116a1d9aa9c5" } modify_external_catalogs: true
I20250625 13:52:10.823091 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4dfad83f70604649b15f116a1d9aa9c5" } modify_external_catalogs: true
W20250625 13:52:10.842967 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d12f07f4145240958d7decf21c03b6bc consensus state has no leader
W20250625 13:52:10.844009 851 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 6c5fc63828d9408b9cf774a966f43cf2
I20250625 13:52:10.844162 851 leader_election.cc:304] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:10.845024 2204 raft_consensus.cc:2747] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:10.848112 1168 ts_tablet_manager.cc:1918] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:10.848635 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
I20250625 13:52:10.849232 751 tablet_service.cc:1515] Processing DeleteTablet for tablet c9ae350a9dbf476081dc56a709de42c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.856575 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 2770194711ec4a86bb1354517b095f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:10.860997 1168 log.cc:1199] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/3b0037d09796455cbaaadfaa9c010673
I20250625 13:52:10.861567 1168 ts_tablet_manager.cc:1939] T 3b0037d09796455cbaaadfaa9c010673 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:10.864085 1166 raft_consensus.cc:357] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.864288 1166 raft_consensus.cc:383] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.864341 1166 raft_consensus.cc:738] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.864521 1166 consensus_queue.cc:260] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.866113 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 791) took 2743 ms. Trace:
I20250625 13:52:10.866714 1168 rpcz_store.cc:276] 0625 13:52:08.122477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.122789 (+ 312us) service_pool.cc:225] Handling call
0625 13:52:10.555775 (+2432986us) ts_tablet_manager.cc:1145] Deleting tablet 3b0037d09796455cbaaadfaa9c010673
0625 13:52:10.717552 (+161777us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.725336 (+ 7784us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:10.848010 (+122674us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:10.866092 (+ 18082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432766}
I20250625 13:52:10.867684 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 3b0037d09796455cbaaadfaa9c010673 (table default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae [id=6f1fa6d1db4e4abaae4d06299152224f]) successfully deleted
I20250625 13:52:10.868616 1168 tablet_replica.cc:331] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:10.869060 1168 raft_consensus.cc:2241] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:10.869539 1168 raft_consensus.cc:2270] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:10.870913 1168 ts_tablet_manager.cc:1905] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:10.873106 1166 ts_tablet_manager.cc:1428] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.099s user 0.001s sys 0.001s
I20250625 13:52:10.873427 1166 tablet_bootstrap.cc:492] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:10.878019 1170 raft_consensus.cc:357] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.878225 1170 raft_consensus.cc:383] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.878288 1170 raft_consensus.cc:738] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.878480 1170 consensus_queue.cc:260] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.878674 1018 tablet_service.cc:1468] Processing CreateTablet for tablet aafe89794fce4a56b50c4a79ded259a7 (DEFAULT_TABLE table=default.Table_0f4d14b4d5d84cab89f1ce9fa6a60e62 [id=86ed36d73e5040c4bcec5679c9b543d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.879000 1170 ts_tablet_manager.cc:1428] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.088s user 0.002s sys 0.000s
I20250625 13:52:10.881461 1166 tablet_bootstrap.cc:654] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:10.886191 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d0f8fc1edba14555a44cee33f16218ee" } modify_external_catalogs: true
I20250625 13:52:10.886435 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d0f8fc1edba14555a44cee33f16218ee" } modify_external_catalogs: true
W20250625 13:52:10.886862 851 leader_election.cc:343] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2404e712e18146b68757bd77362c1358
I20250625 13:52:10.886997 851 leader_election.cc:304] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:10.887594 2192 raft_consensus.cc:2747] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:10.890874 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c9ae350a9dbf476081dc56a709de42c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9ae350a9dbf476081dc56a709de42c8 already in progress: creating tablet
W20250625 13:52:10.891338 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2770194711ec4a86bb1354517b095f60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2770194711ec4a86bb1354517b095f60 already in progress: creating tablet
I20250625 13:52:10.901932 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 413502c39dd04c02b088071a445b69c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.902599 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b350adad81545bc9d5ee61caadc1f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.902776 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00d7323dcae453683206e12ad22e1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.902957 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 170ffbff76af447a83d663fa8688b7a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.903223 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f88091205b944b386094e50cf92c36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.903733 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1e660c5b244b1c954a1c6426fd959e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.905010 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafe89794fce4a56b50c4a79ded259a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.905831 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 63f82f4a24d44888b222c93f5d614ba5 (DEFAULT_TABLE table=default.Table_ea94ac9a52d84ddda29e98fa67d38868 [id=f8d584b01250470c9883d9ee8596d519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.906442 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1c3914808b7547c7851ed153bbbcddb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.906638 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ff57ade42e5b4619bdacc7ccacc36ae9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:10.906958 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a091a655459e4488b9f5d5798673fd42" } modify_external_catalogs: true
I20250625 13:52:10.907142 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a091a655459e4488b9f5d5798673fd42" } modify_external_catalogs: true
W20250625 13:52:10.907243 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff57ade42e5b4619bdacc7ccacc36ae9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.907528 1170 tablet_bootstrap.cc:492] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:10.879048 1011 tablet_service.cc:1468] Processing CreateTablet for tablet c08720d4fafa4784ab5dbab02c725545 (DEFAULT_TABLE table=default.Table_94eb551cf0ef413b871b98aff36fcdf0 [id=9249a07a87fe4c7fbd26e07b9b968082]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.910980 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63f82f4a24d44888b222c93f5d614ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.913590 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d60f5d5746d44a8a8bb4a317d15f306b (DEFAULT_TABLE table=default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:10.915445 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c3914808b7547c7851ed153bbbcddb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.920873 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8e8fabf01ba94ac8b83358459a201156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.921392 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e8fabf01ba94ac8b83358459a201156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.922551 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_29be1cb29e48403cb7e0c22d2d3ab7bd" } new_table_name: "default.Table_7dfba296bba340fa9bd98980113ebfbb" modify_external_catalogs: true
I20250625 13:52:10.933619 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 8d39a96a128943438ead46a70f47c798 (DEFAULT_TABLE table=default.Table_c9744a7c82cf465696fd5127da2131ba [id=862bc721f3cf4ba794a56b6ea8ce53a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.934846 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d39a96a128943438ead46a70f47c798. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.937975 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e2711256c9042ebb651ae0ca3d02c89 consensus state has no leader
I20250625 13:52:10.945240 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 29758dd57b8d49e8869510c41122b97e (DEFAULT_TABLE table=default.Table_1ec436c40a764dbba4243d2522dcfb1e [id=5758233a9d5444f3be39936851eca9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.945451 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet a1a9a69d26c14a62b26e0e90157782af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.947230 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 973c483c87b84ea09b6c7d556e54b490 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.949576 2233 raft_consensus.cc:491] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:10.949705 2233 raft_consensus.cc:513] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:10.953032 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet e3939ef22ea344328721dfba67d933d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.956777 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29758dd57b8d49e8869510c41122b97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.960806 1020 tablet_service.cc:1468] Processing CreateTablet for tablet b55fd2d3609844d5ba9777190bb6e0b4 (DEFAULT_TABLE table=default.Table_f4e7c3d0a0cb4e4a94d0a2e2e63610c4 [id=744f62b816aa44fbbb70c13f46818a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.961491 1026 tablet_service.cc:1468] Processing CreateTablet for tablet fbc14c29aed345c9a26c8964ff094da4 (DEFAULT_TABLE table=default.Table_85855211d1e441d48d2aeb29e80d3dd5 [id=7863d92166ff4ccab85ed4ad79bea409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.963209 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c08720d4fafa4784ab5dbab02c725545. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.964201 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 754c690dd747460eafbd13ea9efdc692 (DEFAULT_TABLE table=default.Table_92b3804731fc429a8b6412d0be16b8ad [id=43af99dd61534e2aaba6c9f442ca34a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.964529 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 754c690dd747460eafbd13ea9efdc692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.967499 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60f5d5746d44a8a8bb4a317d15f306b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.967839 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6fc366fe7a4f4086a2f2cf60731c8089" } new_table_name: "default.Table_2f0f0fd49d3246f9b3ffededc650120a" modify_external_catalogs: true
I20250625 13:52:10.968430 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d12f07f4145240958d7decf21c03b6bc (DEFAULT_TABLE table=default.Table_464af049186b4c6d849a988e68947171 [id=bd1cd6099a8244348047ebb27abf3590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.969476 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbc14c29aed345c9a26c8964ff094da4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.969627 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 499360us
I20250625 13:52:10.971210 1171 tablet_bootstrap.cc:492] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:10.971294 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 2593db823e974f1eb653470d5a23338f (DEFAULT_TABLE table=default.Table_628c945c37c84b88a1ed70deaadd49a6 [id=b4206cbbd39643549beb76e6025e6c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:10.971374 1171 ts_tablet_manager.cc:1397] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.242s user 0.007s sys 0.033s
I20250625 13:52:10.971626 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2593db823e974f1eb653470d5a23338f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.972069 1171 raft_consensus.cc:357] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.972152 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet bd4cc91d274a45a393beffaaa59a35f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.972252 1171 raft_consensus.cc:383] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:10.972318 1171 raft_consensus.cc:738] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:10.972532 1171 consensus_queue.cc:260] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:10.972998 1171 ts_tablet_manager.cc:1428] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:10.973506 1171 tablet_bootstrap.cc:492] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:10.974983 1171 tablet_bootstrap.cc:654] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:10.976343 711 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:10.976469 711 leader_election.cc:304] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:10.976629 711 raft_consensus.cc:2684] T 74884bf313724d678d3b5f5869f09f97 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:52:10.976959 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723246e1e6824faf9f9d39c3ea941fe4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:10.977808 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4acfcd35d6434ee59b9b0d831a59938c consensus state has no leader
I20250625 13:52:10.979936 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3fe8ea719ca94aaca7b19fadc398d91d" } new_table_name: "default.Table_d7faba35e83944d586e1e53d4b524950" modify_external_catalogs: true
I20250625 13:52:10.982623 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ee7a863cc44d4067a1f5a29e6fe267af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:10.983201 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee7a863cc44d4067a1f5a29e6fe267af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:10.987720 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0b1631a468c34b49a62449290f63ea32" } new_table_name: "default.Table_e50eae9bd67942f5a09da3884500dac7" modify_external_catalogs: true
W20250625 13:52:10.989620 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd4142572b344456a0d8248be914ad8f consensus state has no leader
I20250625 13:52:10.990263 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b55fd2d3609844d5ba9777190bb6e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:10.993518 2233 leader_election.cc:290] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:10.994189 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c70919181d4c5392f65df221ec6348 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:10.997375 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12f07f4145240958d7decf21c03b6bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:10.997877 983 leader_election.cc:343] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 3b0037d09796455cbaaadfaa9c010673
W20250625 13:52:11.006028 985 leader_election.cc:343] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3b0037d09796455cbaaadfaa9c010673
I20250625 13:52:11.006186 985 leader_election.cc:304] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:11.006419 2233 raft_consensus.cc:2747] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.007186 1166 tablet_bootstrap.cc:492] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:11.007400 1166 ts_tablet_manager.cc:1397] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.134s user 0.001s sys 0.013s
I20250625 13:52:11.009459 1170 tablet_bootstrap.cc:654] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.009780 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723246e1e6824faf9f9d39c3ea941fe4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:11.011018 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9 consensus state has no leader
I20250625 13:52:11.011981 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4111b95923ed4d73b7eca60b964d9494 (DEFAULT_TABLE table=default.Table_bd3d18582db04d18829da98f82d6ee0c [id=0b427c87bb6446daa95efbaed62b067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.012041 1166 raft_consensus.cc:357] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.012197 1166 raft_consensus.cc:383] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.012253 1166 raft_consensus.cc:738] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:11.012732 849 consensus_peers.cc:487] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:11.012812 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71] alter complete (version 2)
I20250625 13:52:11.013748 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.014395 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 8684eb9104e44227944b21784c54e496 (DEFAULT_TABLE table=default.Table_4d2946a0f06e4f3fa1cde181e94856f3 [id=7bef6cfdaece44549646dc74b108bada]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.014815 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:11.015158 2244 log.cc:927] Time spent T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250625 13:52:11.014869 1166 consensus_queue.cc:260] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.015681 1165 tablet_replica.cc:331] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:11.015854 1165 raft_consensus.cc:2241] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.015983 1165 raft_consensus.cc:2270] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:11.016268 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4111b95923ed4d73b7eca60b964d9494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.016777 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0421a7491bdc4152bf6b1ea0a5b53c9e (DEFAULT_TABLE table=default.Table_15e2aec48b1c4191a1ca7a3b48eac9ab [id=0a0b8545bca14836a8c5f102f328f9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.016914 1165 ts_tablet_manager.cc:1905] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:11.016804 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0b55a0fa335e4435a1f8c85d61d7c13b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.017072 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0421a7491bdc4152bf6b1ea0a5b53c9e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.017304 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b55a0fa335e4435a1f8c85d61d7c13b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.019855 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 30b5b3444fd245bbb2ecd5889d956862 (DEFAULT_TABLE table=default.Table_a3634181d607460c9e84b9cca878d57b [id=f6a1fa8c3e804c6d996be293a1e207ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.020133 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet c9ae350a9dbf476081dc56a709de42c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.020242 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30b5b3444fd245bbb2ecd5889d956862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.020287 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 2770194711ec4a86bb1354517b095f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.021643 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8684eb9104e44227944b21784c54e496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.023411 1165 ts_tablet_manager.cc:1918] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:11.023576 1165 log.cc:1199] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/63d01af6b320489fb99c467c11ed1dc0
I20250625 13:52:11.024158 1165 ts_tablet_manager.cc:1939] T 63d01af6b320489fb99c467c11ed1dc0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:11.025218 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 63d01af6b320489fb99c467c11ed1dc0 (table default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae [id=c4babc967a48425587250196c8d9b4ab]) successfully deleted
I20250625 13:52:11.025619 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9e96f4e40f0c4f01b0f30c1502b3b7b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.025666 1012 tablet_service.cc:1468] Processing CreateTablet for tablet d64d8f832da048e68e688a037b593719 (DEFAULT_TABLE table=default.Table_2e590dac73914afca2846962795030aa [id=ddf2721f40274855986d81f26d332f0b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:11.026115 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e96f4e40f0c4f01b0f30c1502b3b7b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.029702 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f0e1bdf2998341a5988d7e7723151b3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.030211 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0e1bdf2998341a5988d7e7723151b3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.032334 1171 tablet_bootstrap.cc:492] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.032478 1171 ts_tablet_manager.cc:1397] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.009s
I20250625 13:52:11.033212 1171 raft_consensus.cc:357] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.033376 1171 raft_consensus.cc:383] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.033427 1171 raft_consensus.cc:738] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.033641 1171 consensus_queue.cc:260] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.035776 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1e3b96b3cabb46eea9cb5b0275bcda73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.036262 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e3b96b3cabb46eea9cb5b0275bcda73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.037343 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fdada5e5d21042ecb3d4b84dc1f0b299"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.037744 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdada5e5d21042ecb3d4b84dc1f0b299 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.042089 1166 ts_tablet_manager.cc:1428] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20250625 13:52:11.042718 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64d8f832da048e68e688a037b593719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.044420 1166 tablet_bootstrap.cc:492] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:11.044996 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7dfba296bba340fa9bd98980113ebfbb" } new_table_name: "default.Table_8af8a34363004cbaabfa6b6a8371425f" modify_external_catalogs: true
I20250625 13:52:11.045943 1166 tablet_bootstrap.cc:654] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.054443 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0221d3b2f7554e1a9a8159c7fade3b33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.054999 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
W20250625 13:52:11.055056 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0221d3b2f7554e1a9a8159c7fade3b33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.057587 884 tablet_service.cc:1468] Processing CreateTablet for tablet 5bf16bad14734a7c84f559fc18b171e7 (DEFAULT_TABLE table=default.Table_a2251328ba8745ae98b9fead6e8e516e [id=507e2802010849108aa302ab923228c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.057960 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf16bad14734a7c84f559fc18b171e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.059692 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_fe321782d15943acaf71e83adc22ab28 [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 2 got 0
I20250625 13:52:11.065871 894 tablet_service.cc:1468] Processing CreateTablet for tablet 6099af0fd3564a7f979d7235addd336d (DEFAULT_TABLE table=default.Table_639d5431f288425b907989942e4aa7f5 [id=b946aef1cb9f485b88f539ab9a6faaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.066232 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6099af0fd3564a7f979d7235addd336d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.068213 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5bf16bad14734a7c84f559fc18b171e7 (DEFAULT_TABLE table=default.Table_a2251328ba8745ae98b9fead6e8e516e [id=507e2802010849108aa302ab923228c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.068576 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf16bad14734a7c84f559fc18b171e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.069087 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_fe321782d15943acaf71e83adc22ab28 [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 2 got 0
I20250625 13:52:11.071426 756 tablet_service.cc:1468] Processing CreateTablet for tablet af497b011b7248d2b247854b150a8f21 (DEFAULT_TABLE table=default.Table_fd92384a196c4ca1a02f43763f639a77 [id=993f23e6226445e2a7552a6a60936f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.071760 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af497b011b7248d2b247854b150a8f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.072054 879 tablet_service.cc:1468] Processing CreateTablet for tablet af497b011b7248d2b247854b150a8f21 (DEFAULT_TABLE table=default.Table_fd92384a196c4ca1a02f43763f639a77 [id=993f23e6226445e2a7552a6a60936f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.072398 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af497b011b7248d2b247854b150a8f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.074048 2192 raft_consensus.cc:491] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.074168 2192 raft_consensus.cc:513] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.074795 2192 leader_election.cc:290] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:11.075546 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6099af0fd3564a7f979d7235addd336d (DEFAULT_TABLE table=default.Table_639d5431f288425b907989942e4aa7f5 [id=b946aef1cb9f485b88f539ab9a6faaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.075865 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6099af0fd3564a7f979d7235addd336d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.077929 893 tablet_service.cc:1468] Processing CreateTablet for tablet f4f93a7beee14d2892ee387bcc1e76b9 (DEFAULT_TABLE table=default.Table_1c72b998c9314821835754a40d9c88a1 [id=245bae57243f4930a52a93bce8ca48df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.078090 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87f5b36ddb474ed6903c14876b236ba8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:11.078255 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f93a7beee14d2892ee387bcc1e76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.078755 748 tablet_service.cc:1468] Processing CreateTablet for tablet f4f93a7beee14d2892ee387bcc1e76b9 (DEFAULT_TABLE table=default.Table_1c72b998c9314821835754a40d9c88a1 [id=245bae57243f4930a52a93bce8ca48df]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:11.078905 851 leader_election.cc:343] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 87f5b36ddb474ed6903c14876b236ba8
I20250625 13:52:11.079066 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f93a7beee14d2892ee387bcc1e76b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.079173 849 leader_election.cc:343] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.079281 849 leader_election.cc:304] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:11.083456 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:11.083572 747 tablet_service.cc:1468] Processing CreateTablet for tablet 2456662afcc3426c8dbc5e1132c2f3ac (DEFAULT_TABLE table=default.Table_e40aed94ebc340d8b6324a1bd2840e03 [id=b6d49fa4993a47b493d21dcf38131752]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.083910 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2456662afcc3426c8dbc5e1132c2f3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.085641 1171 ts_tablet_manager.cc:1428] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.053s user 0.001s sys 0.001s
I20250625 13:52:11.086364 2192 raft_consensus.cc:2747] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.086962 741 tablet_service.cc:1468] Processing CreateTablet for tablet cebeffb6fa544997af268cc9579aee27 (DEFAULT_TABLE table=default.Table_ec4be031b7ee44a59b7015fda0f6bf44 [id=1a1ec8e7044e414eb6154e505764fc3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.087306 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cebeffb6fa544997af268cc9579aee27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.090430 744 tablet_service.cc:1468] Processing CreateTablet for tablet 972339c91a114f88b8d46233fd2f572d (DEFAULT_TABLE table=default.Table_513b6609ffeb48e592b807215f4e9488 [id=73d0b4eca16b46cb928ef0e4dbd8751f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.090786 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972339c91a114f88b8d46233fd2f572d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.090793 1171 tablet_bootstrap.cc:492] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:11.091231 1014 tablet_service.cc:1468] Processing CreateTablet for tablet f4f93a7beee14d2892ee387bcc1e76b9 (DEFAULT_TABLE table=default.Table_1c72b998c9314821835754a40d9c88a1 [id=245bae57243f4930a52a93bce8ca48df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.096524 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 6099af0fd3564a7f979d7235addd336d (DEFAULT_TABLE table=default.Table_639d5431f288425b907989942e4aa7f5 [id=b946aef1cb9f485b88f539ab9a6faaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.100533 889 tablet_service.cc:1468] Processing CreateTablet for tablet cebeffb6fa544997af268cc9579aee27 (DEFAULT_TABLE table=default.Table_ec4be031b7ee44a59b7015fda0f6bf44 [id=1a1ec8e7044e414eb6154e505764fc3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.100946 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cebeffb6fa544997af268cc9579aee27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.104034 891 tablet_service.cc:1468] Processing CreateTablet for tablet 972339c91a114f88b8d46233fd2f572d (DEFAULT_TABLE table=default.Table_513b6609ffeb48e592b807215f4e9488 [id=73d0b4eca16b46cb928ef0e4dbd8751f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.104429 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972339c91a114f88b8d46233fd2f572d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.108642 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.112334 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 2456662afcc3426c8dbc5e1132c2f3ac (DEFAULT_TABLE table=default.Table_e40aed94ebc340d8b6324a1bd2840e03 [id=b6d49fa4993a47b493d21dcf38131752]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.091974 1016 tablet_service.cc:1468] Processing CreateTablet for tablet af497b011b7248d2b247854b150a8f21 (DEFAULT_TABLE table=default.Table_fd92384a196c4ca1a02f43763f639a77 [id=993f23e6226445e2a7552a6a60936f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.117911 1171 tablet_bootstrap.cc:654] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.121157 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f93a7beee14d2892ee387bcc1e76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.121394 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6099af0fd3564a7f979d7235addd336d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.121575 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2456662afcc3426c8dbc5e1132c2f3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.122296 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 38c954e01db443fc802d76bbbb853c14
I20250625 13:52:11.123416 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 422f4d02482942dba453ff560890137b
I20250625 13:52:11.124711 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 723246e1e6824faf9f9d39c3ea941fe4
W20250625 13:52:11.125888 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e2711256c9042ebb651ae0ca3d02c89 consensus state has no leader
I20250625 13:52:11.128281 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af497b011b7248d2b247854b150a8f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.129593 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 422f4d02482942dba453ff560890137b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.130213 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 38c954e01db443fc802d76bbbb853c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:11.130514 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 422f4d02482942dba453ff560890137b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 422f4d02482942dba453ff560890137b already in progress: creating tablet
I20250625 13:52:11.130679 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 723246e1e6824faf9f9d39c3ea941fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:11.130770 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 38c954e01db443fc802d76bbbb853c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38c954e01db443fc802d76bbbb853c14 already in progress: creating tablet
I20250625 13:52:11.130879 1165 tablet_replica.cc:331] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:11.131068 1165 raft_consensus.cc:2241] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.131213 1165 raft_consensus.cc:2270] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:11.131754 1165 ts_tablet_manager.cc:1905] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:11.132134 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e899e0383341441d9eeca197fd80a8be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.132638 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e899e0383341441d9eeca197fd80a8be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.132539 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 5bf16bad14734a7c84f559fc18b171e7 (DEFAULT_TABLE table=default.Table_a2251328ba8745ae98b9fead6e8e516e [id=507e2802010849108aa302ab923228c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.136104 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7507d50e9f464043b8901491c57391d6
I20250625 13:52:11.136778 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a8334d8187d2473582f0057193874b74
I20250625 13:52:11.138921 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bf16bad14734a7c84f559fc18b171e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.139207 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 7507d50e9f464043b8901491c57391d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.139072 1165 ts_tablet_manager.cc:1918] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:11.140321 1165 log.cc:1199] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/723246e1e6824faf9f9d39c3ea941fe4
I20250625 13:52:11.140647 1165 ts_tablet_manager.cc:1939] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:11.141892 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_931f25deb4904d2fa74ab9437e733369" } new_table_name: "default.Table_963851a3d1c4472eab0949b2c70d4279" modify_external_catalogs: true
I20250625 13:52:11.143352 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 38c954e01db443fc802d76bbbb853c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.143987 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a18783dacb0b4bd09053d2559ed93252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.144630 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a18783dacb0b4bd09053d2559ed93252 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.144726 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 422f4d02482942dba453ff560890137b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.145251 879 tablet_service.cc:1468] Processing CreateTablet for tablet 2456662afcc3426c8dbc5e1132c2f3ac (DEFAULT_TABLE table=default.Table_e40aed94ebc340d8b6324a1bd2840e03 [id=b6d49fa4993a47b493d21dcf38131752]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.145630 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2456662afcc3426c8dbc5e1132c2f3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.146631 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b75460e88d6f4692a0992d1601f9783d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.147171 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b75460e88d6f4692a0992d1601f9783d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.148460 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 723246e1e6824faf9f9d39c3ea941fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.151212 896 tablet_service.cc:1515] Processing DeleteTablet for tablet a8334d8187d2473582f0057193874b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.151501 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 7507d50e9f464043b8901491c57391d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.152437 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a8334d8187d2473582f0057193874b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.152693 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 723246e1e6824faf9f9d39c3ea941fe4 (table default.Table_49756985b3fc428cbc871659e5aabdc6 [id=1178d7027158455bb95716dd9e102ef9]) successfully deleted
W20250625 13:52:11.152937 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7507d50e9f464043b8901491c57391d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7507d50e9f464043b8901491c57391d6 already in progress: creating tablet
W20250625 13:52:11.155005 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a8334d8187d2473582f0057193874b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8334d8187d2473582f0057193874b74 already in progress: creating tablet
W20250625 13:52:11.160620 2247 log.cc:927] Time spent T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250625 13:52:11.160645 1030 tablet_service.cc:1468] Processing CreateTablet for tablet cebeffb6fa544997af268cc9579aee27 (DEFAULT_TABLE table=default.Table_ec4be031b7ee44a59b7015fda0f6bf44 [id=1a1ec8e7044e414eb6154e505764fc3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.161193 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cebeffb6fa544997af268cc9579aee27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.163476 1047 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.2:58031 (request call id 326) took 1034 ms. Trace:
W20250625 13:52:11.163789 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71170aaf26ed4f1b8c458d90db6e2765 consensus state has no leader
I20250625 13:52:11.173027 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:11.179495 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_30935873709947f3adc8b29d8ffc698c" } new_table_name: "default.Table_3280d0e3e7144ed39b5f571ae7707069" modify_external_catalogs: true
W20250625 13:52:11.186111 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34b61a29713e4183ac17e7414807770f consensus state has no leader
I20250625 13:52:11.188227 2192 consensus_queue.cc:1035] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:11.191515 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cc87fc1ee177485192b77fe418e55945" } new_table_name: "default.Table_13065c9f25c443f4b762794456e59b3e" modify_external_catalogs: true
W20250625 13:52:11.197644 711 leader_election.cc:343] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:11.198684 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b493f32299ba42159c594849e0fe39c4 consensus state has no leader
I20250625 13:52:11.198711 1168 ts_tablet_manager.cc:1918] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:11.198830 1168 log.cc:1199] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d43ba17ebfec4244894d1b6e23200c33
I20250625 13:52:11.199149 1168 ts_tablet_manager.cc:1939] T d43ba17ebfec4244894d1b6e23200c33 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:11.199515 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 792) took 3076 ms. Trace:
I20250625 13:52:11.199853 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_01f30a93171b4189adaa5b5c283a43e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.200570 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01f30a93171b4189adaa5b5c283a43e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:11.204188 711 leader_election.cc:343] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.206848 711 leader_election.cc:304] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:11.207170 711 raft_consensus.cc:2684] T 723246e1e6824faf9f9d39c3ea941fe4 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:52:11.212404 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e6b348f960d5446ca1bfe847c5142516" } new_table_name: "default.Table_1b0f553b6f7d471fab8cf8cc41d204c3" modify_external_catalogs: true
W20250625 13:52:11.218463 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf7463962f39468c85fb3dc2edcf608f consensus state has no leader
I20250625 13:52:11.229003 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ee5d5978aaf540b8bc77e7c4bbe55258"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.229597 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee5d5978aaf540b8bc77e7c4bbe55258 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.233081 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:11.239446 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:11.241691 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_76340fa76a5048f9b56482e7e2563551"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.242359 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76340fa76a5048f9b56482e7e2563551 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.245926 1006 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 158) took 1349 ms. Trace:
I20250625 13:52:11.246477 1006 rpcz_store.cc:276] 0625 13:52:09.895752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.895802 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:10.777069 (+881267us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x616000145654. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:11.245626 (+468557us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1937575552}
W20250625 13:52:11.247938 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 43754.4us
I20250625 13:52:11.252249 756 tablet_service.cc:1468] Processing CreateTablet for tablet 835e221fb6fa4e8b83a15fda0f1951af (DEFAULT_TABLE table=default.Table_9e96f4e40f0c4f01b0f30c1502b3b7b4 [id=eb3c5037d8d44b16ac6a2b6e843a79ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.252365 749 tablet_service.cc:1468] Processing CreateTablet for tablet aa555aa6a0124e5bac31c2bb83bb26ae (DEFAULT_TABLE table=default.Table_1c3914808b7547c7851ed153bbbcddb5 [id=7b93e36e077a4c48adde8f2779ea0aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.252754 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 835e221fb6fa4e8b83a15fda0f1951af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.252970 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa555aa6a0124e5bac31c2bb83bb26ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.252689 747 tablet_service.cc:1468] Processing CreateTablet for tablet 61ecf99b86cf4fe1bfac2b0907ef8662 (DEFAULT_TABLE table=default.Table_8e8fabf01ba94ac8b83358459a201156 [id=09be3b3816c641e9b0f5b7b1449c114b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.255568 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ecf99b86cf4fe1bfac2b0907ef8662. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.255354 1008 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 157) took 1375 ms. Trace:
I20250625 13:52:11.255786 1008 rpcz_store.cc:276] 0625 13:52:09.879674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.879742 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:10.778410 (+898668us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x616000145654. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:11.255062 (+476652us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2049874688}
I20250625 13:52:11.258111 2233 raft_consensus.cc:491] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.258255 2233 raft_consensus.cc:513] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:11.258831 2233 leader_election.cc:290] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:11.240824 1166 tablet_bootstrap.cc:492] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:11.259931 1166 ts_tablet_manager.cc:1397] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.216s user 0.002s sys 0.001s
I20250625 13:52:11.260715 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5d467dbf133147e7a517c6fc4dff433a (DEFAULT_TABLE table=default.Table_ff57ade42e5b4619bdacc7ccacc36ae9 [id=5a006a6137de44cb8b40c242c9667088]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.261334 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d467dbf133147e7a517c6fc4dff433a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.262583 985 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 6dd84b8afa2548fe8ce5925f0bf806f0
I20250625 13:52:11.262602 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5d17ee811c9c4e2dbfaae512da4b46f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.263139 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d17ee811c9c4e2dbfaae512da4b46f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.267699 1168 rpcz_store.cc:276] 0625 13:52:08.122602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.123054 (+ 452us) service_pool.cc:225] Handling call
0625 13:52:10.868548 (+2745494us) ts_tablet_manager.cc:1145] Deleting tablet d43ba17ebfec4244894d1b6e23200c33
0625 13:52:10.868558 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:10.938761 (+ 70203us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:11.198584 (+259823us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:11.199498 (+ 914us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2745380}
I20250625 13:52:11.268249 1168 tablet_replica.cc:331] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:11.268468 1168 raft_consensus.cc:2241] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.268646 1168 raft_consensus.cc:2270] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:11.269236 1168 ts_tablet_manager.cc:1905] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:11.270622 748 tablet_service.cc:1468] Processing CreateTablet for tablet 47260cc860264fa8a4082a40e618832f (DEFAULT_TABLE table=default.Table_6d08523c84e84dd487355d82976fc3c8 [id=56cb37467a4642bfbc067aff7adb7c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.270991 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47260cc860264fa8a4082a40e618832f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.272459 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet d43ba17ebfec4244894d1b6e23200c33 (table default.Table_bc6c1303fba14b97b873dd23a423bf8c [id=999e931fc6724eec9b6d8bb94bd3cf23]) successfully deleted
I20250625 13:52:11.272953 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5d467dbf133147e7a517c6fc4dff433a (DEFAULT_TABLE table=default.Table_ff57ade42e5b4619bdacc7ccacc36ae9 [id=5a006a6137de44cb8b40c242c9667088]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.273449 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d467dbf133147e7a517c6fc4dff433a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.273373 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ca0d7155da664e88b759e29a7c5d554b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.273890 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca0d7155da664e88b759e29a7c5d554b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.275564 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:11.208611 2253 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:11.251657 889 tablet_service.cc:1468] Processing CreateTablet for tablet 835e221fb6fa4e8b83a15fda0f1951af (DEFAULT_TABLE table=default.Table_9e96f4e40f0c4f01b0f30c1502b3b7b4 [id=eb3c5037d8d44b16ac6a2b6e843a79ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.279816 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 835e221fb6fa4e8b83a15fda0f1951af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.252764 891 tablet_service.cc:1468] Processing CreateTablet for tablet aa555aa6a0124e5bac31c2bb83bb26ae (DEFAULT_TABLE table=default.Table_1c3914808b7547c7851ed153bbbcddb5 [id=7b93e36e077a4c48adde8f2779ea0aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.284138 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa555aa6a0124e5bac31c2bb83bb26ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.285101 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:11.289503 749 tablet_service.cc:1468] Processing CreateTablet for tablet d861dc6bff5d4b03a9499e91fd22430d (DEFAULT_TABLE table=default.Table_52de821c06b24eea87c7839e7417a783 [id=4b8fad3e0cfd4e0a8d68946deb48a28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.290012 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d861dc6bff5d4b03a9499e91fd22430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.296800 747 tablet_service.cc:1468] Processing CreateTablet for tablet 38b757f6ac764c22b4217145c9553ab6 (DEFAULT_TABLE table=default.Table_f526b40a96af4910ab95446d509bf95b [id=bd28b5c3051b43649a03759f502e5561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.297137 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b757f6ac764c22b4217145c9553ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.299499 748 tablet_service.cc:1468] Processing CreateTablet for tablet a6b2f861d63d4299b3e472cf705a1b67 (DEFAULT_TABLE table=default.Table_876b0fe22f3147528bd9aedba430dc9f [id=8ce01ab8558a4912bf51621a930c092b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.299811 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b2f861d63d4299b3e472cf705a1b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.259698 884 tablet_service.cc:1468] Processing CreateTablet for tablet 61ecf99b86cf4fe1bfac2b0907ef8662 (DEFAULT_TABLE table=default.Table_8e8fabf01ba94ac8b83358459a201156 [id=09be3b3816c641e9b0f5b7b1449c114b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.303388 1171 tablet_bootstrap.cc:492] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.303529 1171 ts_tablet_manager.cc:1397] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.213s user 0.018s sys 0.027s
I20250625 13:52:11.304248 1171 raft_consensus.cc:357] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.304420 1171 raft_consensus.cc:383] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.304469 1171 raft_consensus.cc:738] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.304672 1171 consensus_queue.cc:260] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.307574 741 tablet_service.cc:1468] Processing CreateTablet for tablet 41c0da55e5414976902467bd2f550037 (DEFAULT_TABLE table=default.Table_36b5354e955b4364845c9fbb1e93d0eb [id=3162c98fa7b3456e857237ae5f01756a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.307709 1171 ts_tablet_manager.cc:1428] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:11.307921 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c0da55e5414976902467bd2f550037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.307971 1171 tablet_bootstrap.cc:492] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:11.309911 1171 tablet_bootstrap.cc:654] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.311969 894 tablet_service.cc:1468] Processing CreateTablet for tablet 47260cc860264fa8a4082a40e618832f (DEFAULT_TABLE table=default.Table_6d08523c84e84dd487355d82976fc3c8 [id=56cb37467a4642bfbc067aff7adb7c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.312399 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47260cc860264fa8a4082a40e618832f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.314327 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd78a52e6604affb896881e05d26957 (DEFAULT_TABLE table=default.Table_3765a1c34b154d1f97898bad646c9ecf [id=1487a09c7ddd44bfb56c1cfbbf73be5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.314651 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd78a52e6604affb896881e05d26957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.316040 748 tablet_service.cc:1468] Processing CreateTablet for tablet d83473ba3ebd4843b7208b8f5935835c (DEFAULT_TABLE table=default.Table_ed63056903cc48609558b9b8419c6c8b [id=113def9728144b108b7485ade091ff9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.316399 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83473ba3ebd4843b7208b8f5935835c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.320343 896 tablet_service.cc:1468] Processing CreateTablet for tablet d861dc6bff5d4b03a9499e91fd22430d (DEFAULT_TABLE table=default.Table_52de821c06b24eea87c7839e7417a783 [id=4b8fad3e0cfd4e0a8d68946deb48a28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.320770 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d861dc6bff5d4b03a9499e91fd22430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.317734 741 tablet_service.cc:1468] Processing CreateTablet for tablet 12a7db105cb34bb2ac7a411256c43cae (DEFAULT_TABLE table=default.Table_065463f5287f4182ace6007e8cfcec03 [id=915767e7f591429fa2b5a950c7a60f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.323607 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7db105cb34bb2ac7a411256c43cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.324164 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_065463f5287f4182ace6007e8cfcec03" } modify_external_catalogs: true
I20250625 13:52:11.324402 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_065463f5287f4182ace6007e8cfcec03" } modify_external_catalogs: true
I20250625 13:52:11.325225 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_30fbf695c7684ae6a586263957012855" } modify_external_catalogs: true
I20250625 13:52:11.325371 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_30fbf695c7684ae6a586263957012855" } modify_external_catalogs: true
I20250625 13:52:11.328321 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c798e33a708c4d87b816a43d3a9d2d2b" } modify_external_catalogs: true
I20250625 13:52:11.328493 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c798e33a708c4d87b816a43d3a9d2d2b" } modify_external_catalogs: true
I20250625 13:52:11.329664 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_49a5b264518840179315082ba4922b6c" } new_table_name: "default.Table_01c5150924d84a429e9004bb668e4550" modify_external_catalogs: true
I20250625 13:52:11.329954 890 tablet_service.cc:1468] Processing CreateTablet for tablet a6b2f861d63d4299b3e472cf705a1b67 (DEFAULT_TABLE table=default.Table_876b0fe22f3147528bd9aedba430dc9f [id=8ce01ab8558a4912bf51621a930c092b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.330379 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b2f861d63d4299b3e472cf705a1b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.336966 1171 tablet_bootstrap.cc:492] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.337657 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2f0f0fd49d3246f9b3ffededc650120a" } modify_external_catalogs: true
I20250625 13:52:11.337819 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2f0f0fd49d3246f9b3ffededc650120a" } modify_external_catalogs: true
I20250625 13:52:11.341059 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet af704cdf44ef42f7b5ea10b199d0f017
I20250625 13:52:11.341804 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5d2ce669ecd441939be8e8c71eea7bfa
I20250625 13:52:11.342789 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d2ce669ecd441939be8e8c71eea7bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:11.343416 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5d2ce669ecd441939be8e8c71eea7bfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2ce669ecd441939be8e8c71eea7bfa already in progress: creating tablet
I20250625 13:52:11.339067 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_65fb980f16aa4d5d9aa270260fb1b896" } modify_external_catalogs: true
I20250625 13:52:11.343917 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_65fb980f16aa4d5d9aa270260fb1b896" } modify_external_catalogs: true
W20250625 13:52:11.346333 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 027b1281543b4d559461e8d222080c30 consensus state has no leader
I20250625 13:52:11.346920 744 tablet_service.cc:1468] Processing CreateTablet for tablet bb993cc5648f44a6bf8c15544fff7cca (DEFAULT_TABLE table=default.Table_ee7a863cc44d4067a1f5a29e6fe267af [id=28eab56011a1491abfc13e26d286d5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.347286 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb993cc5648f44a6bf8c15544fff7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.349439 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_521bb02a0fa64038b747db4f73fa8c90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.349906 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_521bb02a0fa64038b747db4f73fa8c90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.352216 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 12a7db105cb34bb2ac7a411256c43cae
I20250625 13:52:11.353245 747 tablet_service.cc:1515] Processing DeleteTablet for tablet af704cdf44ef42f7b5ea10b199d0f017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.337360 749 tablet_service.cc:1468] Processing CreateTablet for tablet cf6c3973912b4c5fbec28b60c4fdb408 (DEFAULT_TABLE table=default.Table_1e3b96b3cabb46eea9cb5b0275bcda73 [id=3c6db0d6e3fb4314ba1dddf8654c2386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.353464 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 972339c91a114f88b8d46233fd2f572d (DEFAULT_TABLE table=default.Table_513b6609ffeb48e592b807215f4e9488 [id=73d0b4eca16b46cb928ef0e4dbd8751f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.353567 1171 ts_tablet_manager.cc:1397] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
I20250625 13:52:11.353770 877 tablet_service.cc:1468] Processing CreateTablet for tablet 12a7db105cb34bb2ac7a411256c43cae (DEFAULT_TABLE table=default.Table_065463f5287f4182ace6007e8cfcec03 [id=915767e7f591429fa2b5a950c7a60f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.353848 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 972339c91a114f88b8d46233fd2f572d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.354105 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7db105cb34bb2ac7a411256c43cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.354369 1171 raft_consensus.cc:357] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.354507 1171 raft_consensus.cc:383] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.354565 1171 raft_consensus.cc:738] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:52:11.354979 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet af704cdf44ef42f7b5ea10b199d0f017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af704cdf44ef42f7b5ea10b199d0f017 already in progress: creating tablet
I20250625 13:52:11.355242 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7db105cb34bb2ac7a411256c43cae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.355823 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_afac01f597474109bbc2f29c5279b1fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.355924 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 12a7db105cb34bb2ac7a411256c43cae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12a7db105cb34bb2ac7a411256c43cae already in progress: creating tablet
W20250625 13:52:11.356385 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afac01f597474109bbc2f29c5279b1fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.356801 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6c3973912b4c5fbec28b60c4fdb408. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.358168 1171 consensus_queue.cc:260] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.358652 1171 ts_tablet_manager.cc:1428] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:11.360754 1171 tablet_bootstrap.cc:492] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:11.362061 1171 tablet_bootstrap.cc:654] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.163563 1047 rpcz_store.cc:276] 0625 13:52:10.128896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.128989 (+ 93us) service_pool.cc:225] Handling call
0625 13:52:10.129166 (+ 177us) raft_consensus.cc:1469] Updating replica for 1 ops
0625 13:52:10.129290 (+ 124us) raft_consensus.cc:1532] Early marking committed up to index 0
0625 13:52:10.129297 (+ 7us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0625 13:52:10.134553 (+ 5256us) alter_schema_op.cc:128] START. Timestamp: 1750859530078996
0625 13:52:10.763919 (+629366us) log.cc:844] Serialized 173 byte log entry
0625 13:52:11.107011 (+343092us) raft_consensus.cc:1643] Marking committed up to 1
0625 13:52:11.107375 (+ 364us) raft_consensus.cc:1695] Filling consensus response to leader.
0625 13:52:11.108044 (+ 669us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0625 13:52:11.163375 (+ 55331us) raft_consensus.cc:1683] finished
0625 13:52:11.163393 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0625 13:52:11.163455 (+ 62us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:52:10.763828 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0625 13:52:10.809067 (+ 45239us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0625 13:52:10.809079 (+ 12us) alter_schema_op.cc:66] Acquired schema lock
0625 13:52:10.809086 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
Metrics: {"thread_start_us":341012,"threads_started":1,"wal-append.queue_time_us":309401,"child_traces":[["op",{"prepare.queue_time_us":577845,"prepare.run_cpu_time_us":269,"prepare.run_wall_time_us":65773,"thread_start_us":629029,"threads_started":1}]]}
I20250625 13:52:11.368168 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 38c954e01db443fc802d76bbbb853c14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.369084 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2a80a6c4ae994eda8c3f3fb7b126616c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.369704 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a80a6c4ae994eda8c3f3fb7b126616c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.370085 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d5806194428c4cb9927a09061a2d6533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.370604 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5806194428c4cb9927a09061a2d6533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.338244 896 tablet_service.cc:1468] Processing CreateTablet for tablet 38b757f6ac764c22b4217145c9553ab6 (DEFAULT_TABLE table=default.Table_f526b40a96af4910ab95446d509bf95b [id=bd28b5c3051b43649a03759f502e5561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.339833 888 tablet_service.cc:1468] Processing CreateTablet for tablet d83473ba3ebd4843b7208b8f5935835c (DEFAULT_TABLE table=default.Table_ed63056903cc48609558b9b8419c6c8b [id=113def9728144b108b7485ade091ff9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.371166 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83473ba3ebd4843b7208b8f5935835c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.372088 743 tablet_service.cc:1468] Processing CreateTablet for tablet fee74694c02b4fa2a405c57ae7c22a95 (DEFAULT_TABLE table=default.Table_6b8809b44d10450497c7a9f223dda4de [id=a90f9e531c534670917422e193674d1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.372476 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee74694c02b4fa2a405c57ae7c22a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.375344 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0a8b5ac0e4a447a6b313a397677475fa" } new_table_name: "default.Table_fa9ad06641b742c9941821140bb9aa35" modify_external_catalogs: true
I20250625 13:52:11.375416 744 tablet_service.cc:1468] Processing CreateTablet for tablet 97272adfbe254967b099a597ca60d464 (DEFAULT_TABLE table=default.Table_220446a0ea6a427d9940bdb6fd32dad7 [id=ddde40591af64880ad0321b8b180f7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.375780 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97272adfbe254967b099a597ca60d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.340202 890 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd78a52e6604affb896881e05d26957 (DEFAULT_TABLE table=default.Table_3765a1c34b154d1f97898bad646c9ecf [id=1487a09c7ddd44bfb56c1cfbbf73be5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.341467 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.377956 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd78a52e6604affb896881e05d26957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.369554 2251 tablet.cc:1722] T dc75a0a8421b457089cc707d9eb6b76f P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:11.379606 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_943353535aac4d8085a467fcadb1af36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.380923 747 tablet_service.cc:1468] Processing CreateTablet for tablet 4692e7df2014475c86de7ed7735ef3c0 (DEFAULT_TABLE table=default.Table_0b55a0fa335e4435a1f8c85d61d7c13b [id=4a377b59e94f408c871c5a1839d18e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.381701 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b757f6ac764c22b4217145c9553ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.382073 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4692e7df2014475c86de7ed7735ef3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.385926 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 422f4d02482942dba453ff560890137b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.387145 741 tablet_service.cc:1468] Processing CreateTablet for tablet 2e4289f76f754f39ad1fcdb6c2cc7d87 (DEFAULT_TABLE table=default.Table_f0e1bdf2998341a5988d7e7723151b3d [id=0eb88cded43b4b1fb0a745e1d66803cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.262684 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:11.348202 883 tablet_service.cc:1468] Processing CreateTablet for tablet 41c0da55e5414976902467bd2f550037 (DEFAULT_TABLE table=default.Table_36b5354e955b4364845c9fbb1e93d0eb [id=3162c98fa7b3456e857237ae5f01756a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.387480 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4289f76f754f39ad1fcdb6c2cc7d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.387595 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c0da55e5414976902467bd2f550037. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.392745 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_943353535aac4d8085a467fcadb1af36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.393122 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_53926d5babb045a4854877f904b639ec" } new_table_name: "default.Table_12e1df9c1f1c49e8bc2a34c01de01d13" modify_external_catalogs: true
I20250625 13:52:11.393251 912 raft_consensus.cc:2391] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in this term.
I20250625 13:52:11.395648 983 leader_election.cc:304] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:11.396028 2233 raft_consensus.cc:3058] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:11.399370 2233 raft_consensus.cc:2747] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.400055 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet a8334d8187d2473582f0057193874b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.401847 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 7507d50e9f464043b8901491c57391d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.402626 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 835e221fb6fa4e8b83a15fda0f1951af (DEFAULT_TABLE table=default.Table_9e96f4e40f0c4f01b0f30c1502b3b7b4 [id=eb3c5037d8d44b16ac6a2b6e843a79ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.403208 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 835e221fb6fa4e8b83a15fda0f1951af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.404069 883 tablet_service.cc:1468] Processing CreateTablet for tablet 2e4289f76f754f39ad1fcdb6c2cc7d87 (DEFAULT_TABLE table=default.Table_f0e1bdf2998341a5988d7e7723151b3d [id=0eb88cded43b4b1fb0a745e1d66803cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.404423 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4289f76f754f39ad1fcdb6c2cc7d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.406013 1015 tablet_service.cc:1468] Processing CreateTablet for tablet aa555aa6a0124e5bac31c2bb83bb26ae (DEFAULT_TABLE table=default.Table_1c3914808b7547c7851ed153bbbcddb5 [id=7b93e36e077a4c48adde8f2779ea0aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.406368 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa555aa6a0124e5bac31c2bb83bb26ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.408102 895 tablet_service.cc:1468] Processing CreateTablet for tablet cf6c3973912b4c5fbec28b60c4fdb408 (DEFAULT_TABLE table=default.Table_1e3b96b3cabb46eea9cb5b0275bcda73 [id=3c6db0d6e3fb4314ba1dddf8654c2386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.408474 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6c3973912b4c5fbec28b60c4fdb408. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.393134 2251 tablet.cc:1722] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:11.412513 2192 raft_consensus.cc:491] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.412645 2192 raft_consensus.cc:513] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.413242 2192 leader_election.cc:290] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:11.413878 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4b61d4f7f0474aac9b46c532f6f403a0
I20250625 13:52:11.414655 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5e23db69da0c4ecd974b3ce372f254b7" } new_table_name: "default.Table_14ec8e2e12ed48ebad0ef4224a49e058" modify_external_catalogs: true
W20250625 13:52:11.415355 851 leader_election.cc:343] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:11.415818 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 723246e1e6824faf9f9d39c3ea941fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.418217 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4acfcd35d6434ee59b9b0d831a59938c
I20250625 13:52:11.418939 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3f19c3d36661484dac5b43b45c74851c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.419572 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f19c3d36661484dac5b43b45c74851c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.419971 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d2ce669ecd441939be8e8c71eea7bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.420480 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7db105cb34bb2ac7a411256c43cae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.420652 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 61ecf99b86cf4fe1bfac2b0907ef8662 (DEFAULT_TABLE table=default.Table_8e8fabf01ba94ac8b83358459a201156 [id=09be3b3816c641e9b0f5b7b1449c114b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.420933 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 74884bf313724d678d3b5f5869f09f97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.421257 1166 raft_consensus.cc:357] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.421505 1166 raft_consensus.cc:383] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.421633 1166 raft_consensus.cc:738] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.421902 1166 consensus_queue.cc:260] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.424500 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0c326adff3bb4c6ea214f62d5a9c3e86" } new_table_name: "default.Table_4ef4bb15e00b423ba18232d0feb46e36" modify_external_catalogs: true
I20250625 13:52:11.427464 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "042359eacaa2451abc0fa6ed447a2905" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:11.428015 849 leader_election.cc:343] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.428138 849 leader_election.cc:304] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:11.428684 2192 raft_consensus.cc:2747] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.429100 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ecf99b86cf4fe1bfac2b0907ef8662. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.431967 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 4acfcd35d6434ee59b9b0d831a59938c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.436741 1022 tablet_service.cc:1468] Processing CreateTablet for tablet cf6c3973912b4c5fbec28b60c4fdb408 (DEFAULT_TABLE table=default.Table_1e3b96b3cabb46eea9cb5b0275bcda73 [id=3c6db0d6e3fb4314ba1dddf8654c2386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.437182 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6c3973912b4c5fbec28b60c4fdb408. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.437786 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7a3b5892cacf49f29167f2125d14f34a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.438400 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet af704cdf44ef42f7b5ea10b199d0f017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.438372 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 0fd78a52e6604affb896881e05d26957 (DEFAULT_TABLE table=default.Table_3765a1c34b154d1f97898bad646c9ecf [id=1487a09c7ddd44bfb56c1cfbbf73be5c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:11.438421 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a3b5892cacf49f29167f2125d14f34a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.438699 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd78a52e6604affb896881e05d26957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.439005 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 41c0da55e5414976902467bd2f550037 (DEFAULT_TABLE table=default.Table_36b5354e955b4364845c9fbb1e93d0eb [id=3162c98fa7b3456e857237ae5f01756a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.439291 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c0da55e5414976902467bd2f550037. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.439363 1166 ts_tablet_manager.cc:1428] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.179s user 0.001s sys 0.001s
I20250625 13:52:11.439600 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 2e4289f76f754f39ad1fcdb6c2cc7d87 (DEFAULT_TABLE table=default.Table_f0e1bdf2998341a5988d7e7723151b3d [id=0eb88cded43b4b1fb0a745e1d66803cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.439877 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4289f76f754f39ad1fcdb6c2cc7d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.441185 1166 tablet_bootstrap.cc:492] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:11.442085 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4acfcd35d6434ee59b9b0d831a59938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4acfcd35d6434ee59b9b0d831a59938c already in progress: creating tablet
I20250625 13:52:11.442333 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b61d4f7f0474aac9b46c532f6f403a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.442389 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.443938 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a6b2f861d63d4299b3e472cf705a1b67 (DEFAULT_TABLE table=default.Table_876b0fe22f3147528bd9aedba430dc9f [id=8ce01ab8558a4912bf51621a930c092b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.444283 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b2f861d63d4299b3e472cf705a1b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.445420 1165 tablet_replica.cc:331] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:11.445596 1165 raft_consensus.cc:2241] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.445740 1165 raft_consensus.cc:2270] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:11.448318 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4b61d4f7f0474aac9b46c532f6f403a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b61d4f7f0474aac9b46c532f6f403a0 already in progress: creating tablet
I20250625 13:52:11.449270 1165 ts_tablet_manager.cc:1905] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:11.452193 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d2ce669ecd441939be8e8c71eea7bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.452642 1030 tablet_service.cc:1468] Processing CreateTablet for tablet d83473ba3ebd4843b7208b8f5935835c (DEFAULT_TABLE table=default.Table_ed63056903cc48609558b9b8419c6c8b [id=113def9728144b108b7485ade091ff9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.453022 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d83473ba3ebd4843b7208b8f5935835c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.453809 1019 tablet_service.cc:1468] Processing CreateTablet for tablet bb993cc5648f44a6bf8c15544fff7cca (DEFAULT_TABLE table=default.Table_ee7a863cc44d4067a1f5a29e6fe267af [id=28eab56011a1491abfc13e26d286d5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.454133 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb993cc5648f44a6bf8c15544fff7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.454144 1171 tablet_bootstrap.cc:492] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.455513 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 12a7db105cb34bb2ac7a411256c43cae (DEFAULT_TABLE table=default.Table_065463f5287f4182ace6007e8cfcec03 [id=915767e7f591429fa2b5a950c7a60f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.455814 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7db105cb34bb2ac7a411256c43cae. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.459739 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a0b94879874446b569b3cb3250b185 consensus state has no leader
I20250625 13:52:11.459796 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 38b757f6ac764c22b4217145c9553ab6 (DEFAULT_TABLE table=default.Table_f526b40a96af4910ab95446d509bf95b [id=bd28b5c3051b43649a03759f502e5561]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.460134 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b757f6ac764c22b4217145c9553ab6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.460441 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2e6fbf3d81547f798b246a8b8f61387 consensus state has no leader
I20250625 13:52:11.460698 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7db105cb34bb2ac7a411256c43cae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:11.461453 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc350f8573f74dd88dcbc4c9a7c192c4 consensus state has no leader
W20250625 13:52:11.462198 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61542e4ef6a04dbb8a1b989a0767507a consensus state has no leader
I20250625 13:52:11.465277 1171 ts_tablet_manager.cc:1397] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.003s
I20250625 13:52:11.466424 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4215c08737a746b59dd7eb42a6a34ba5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.467144 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4215c08737a746b59dd7eb42a6a34ba5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.467500 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d861dc6bff5d4b03a9499e91fd22430d (DEFAULT_TABLE table=default.Table_52de821c06b24eea87c7839e7417a783 [id=4b8fad3e0cfd4e0a8d68946deb48a28f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.467844 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d861dc6bff5d4b03a9499e91fd22430d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.468302 1165 ts_tablet_manager.cc:1918] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:11.468410 1165 log.cc:1199] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/70959ec681524518a7ec2c9ed4016ad1
I20250625 13:52:11.468536 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 5d467dbf133147e7a517c6fc4dff433a (DEFAULT_TABLE table=default.Table_ff57ade42e5b4619bdacc7ccacc36ae9 [id=5a006a6137de44cb8b40c242c9667088]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.468727 1165 ts_tablet_manager.cc:1939] T 70959ec681524518a7ec2c9ed4016ad1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:11.468860 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d467dbf133147e7a517c6fc4dff433a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.469514 2192 raft_consensus.cc:491] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.469667 2192 raft_consensus.cc:513] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.470449 2192 leader_election.cc:290] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:11.470783 1171 raft_consensus.cc:357] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.470973 1171 raft_consensus.cc:383] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.471025 1171 raft_consensus.cc:738] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.471207 1171 consensus_queue.cc:260] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.471408 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 70959ec681524518a7ec2c9ed4016ad1 (table default.Table_3df36037c9584e4c94d313cdf1402fdd [id=d1dc28e680ae47fe8b0bce895330e353]) successfully deleted
I20250625 13:52:11.471695 1171 ts_tablet_manager.cc:1428] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:52:11.471976 1171 tablet_bootstrap.cc:492] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:11.471942 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74884bf313724d678d3b5f5869f09f97" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:11.472368 851 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 74884bf313724d678d3b5f5869f09f97
W20250625 13:52:11.472792 849 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.472908 849 leader_election.cc:304] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:11.473289 2210 raft_consensus.cc:2747] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.473419 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7c5253f2510845c58dc974a5a0840b62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.473924 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c5253f2510845c58dc974a5a0840b62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.476577 1171 tablet_bootstrap.cc:654] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.477375 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ae5f1d414794474b2178b431cbc1d44 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:11.485445 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0977431f341c4a25825dd7be66a01913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.486136 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0977431f341c4a25825dd7be66a01913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.496556 879 tablet_service.cc:1468] Processing CreateTablet for tablet 97272adfbe254967b099a597ca60d464 (DEFAULT_TABLE table=default.Table_220446a0ea6a427d9940bdb6fd32dad7 [id=ddde40591af64880ad0321b8b180f7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.499780 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97272adfbe254967b099a597ca60d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.504262 895 tablet_service.cc:1468] Processing CreateTablet for tablet fee74694c02b4fa2a405c57ae7c22a95 (DEFAULT_TABLE table=default.Table_6b8809b44d10450497c7a9f223dda4de [id=a90f9e531c534670917422e193674d1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.504784 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee74694c02b4fa2a405c57ae7c22a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.511718 894 tablet_service.cc:1468] Processing CreateTablet for tablet 4692e7df2014475c86de7ed7735ef3c0 (DEFAULT_TABLE table=default.Table_0b55a0fa335e4435a1f8c85d61d7c13b [id=4a377b59e94f408c871c5a1839d18e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.512182 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4692e7df2014475c86de7ed7735ef3c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.513473 1171 tablet_bootstrap.cc:492] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.513610 1171 ts_tablet_manager.cc:1397] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.006s
I20250625 13:52:11.514288 1171 raft_consensus.cc:357] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.514449 1171 raft_consensus.cc:383] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.514496 1171 raft_consensus.cc:738] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.514686 1171 consensus_queue.cc:260] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.515115 1171 ts_tablet_manager.cc:1428] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 13:52:11.515251 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 47260cc860264fa8a4082a40e618832f (DEFAULT_TABLE table=default.Table_6d08523c84e84dd487355d82976fc3c8 [id=56cb37467a4642bfbc067aff7adb7c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.515697 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47260cc860264fa8a4082a40e618832f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.521608 883 tablet_service.cc:1515] Processing DeleteTablet for tablet af704cdf44ef42f7b5ea10b199d0f017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.524806 2210 raft_consensus.cc:491] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.524946 2210 raft_consensus.cc:513] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:11.525534 2210 leader_election.cc:290] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:11.526335 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:11.526456 747 tablet_service.cc:1468] Processing CreateTablet for tablet 05f205733f3d4c81b9e71709c7559660 (DEFAULT_TABLE table=default.Table_ca0d7155da664e88b759e29a7c5d554b [id=6ad809b2835543f7987c9d35b948a21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.526892 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f205733f3d4c81b9e71709c7559660. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.526935 849 leader_election.cc:343] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.527657 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:11.527905 776 raft_consensus.cc:2466] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:11.528337 851 leader_election.cc:304] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:11.528851 2210 raft_consensus.cc:2802] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:11.528940 2210 raft_consensus.cc:491] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:11.528998 2210 raft_consensus.cc:3058] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:11.531759 2210 raft_consensus.cc:513] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:11.532505 2210 leader_election.cc:290] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:11.532867 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0bebb74334374efa90898ab97a13d509 (DEFAULT_TABLE table=default.Table_5d17ee811c9c4e2dbfaae512da4b46f6 [id=b429e1ee7cdc42ed9deff035bca5e9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.532982 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:11.533288 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bebb74334374efa90898ab97a13d509. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.533435 849 leader_election.cc:343] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.534330 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:11.534518 776 raft_consensus.cc:3058] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:11.535640 741 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd0d216fb394f31afa4478ad2097fe5 (DEFAULT_TABLE table=default.Table_fdada5e5d21042ecb3d4b84dc1f0b299 [id=9cf4473ffa5e415b872597528f5aea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.536042 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd0d216fb394f31afa4478ad2097fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.537423 776 raft_consensus.cc:2466] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:11.537813 851 leader_election.cc:304] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:11.538187 2210 raft_consensus.cc:2802] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:11.538357 2210 raft_consensus.cc:695] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:11.538594 2210 consensus_queue.cc:237] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:11.539594 744 tablet_service.cc:1468] Processing CreateTablet for tablet 3a6220e6861d4f8099258fbfabaf7da8 (DEFAULT_TABLE table=default.Table_b75460e88d6f4692a0992d1601f9783d [id=c9395724c7eb4ab59a5e72d7906c624f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.539996 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6220e6861d4f8099258fbfabaf7da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.542778 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 4acfcd35d6434ee59b9b0d831a59938c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.543540 756 tablet_service.cc:1468] Processing CreateTablet for tablet 78c79a2f6544493ba3f3ec20ea67fa95 (DEFAULT_TABLE table=default.Table_e899e0383341441d9eeca197fd80a8be [id=425d7c5e4b0b4fb69c3b61fc1ca233fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.543893 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6c89a899837f40a8ac9eed9f1d68e420 (DEFAULT_TABLE table=default.Table_a18783dacb0b4bd09053d2559ed93252 [id=6c7502c2b02840f7a7f8c8be17b750d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.544234 743 tablet_service.cc:1468] Processing CreateTablet for tablet 1503714b43c54b738a034ca7d0a8a602 (DEFAULT_TABLE table=default.Table_01f30a93171b4189adaa5b5c283a43e1 [id=13a2c478d08d4f3d80332697879d54d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.544587 750 tablet_service.cc:1468] Processing CreateTablet for tablet 19f1050a29b540558aaf1e31118e4531 (DEFAULT_TABLE table=default.Table_76340fa76a5048f9b56482e7e2563551 [id=0ecf1056ada74055bf862e9b3e86ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.544919 742 tablet_service.cc:1468] Processing CreateTablet for tablet ba762a8edb8847368103e5bede9e9d0b (DEFAULT_TABLE table=default.Table_ee5d5978aaf540b8bc77e7c4bbe55258 [id=9c5c587ce78d469ca11f308fd71a6aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.544900 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c640e0bdbd3e4274ab75511cebd6828d" } modify_external_catalogs: true
I20250625 13:52:11.545097 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c640e0bdbd3e4274ab75511cebd6828d" } modify_external_catalogs: true
I20250625 13:52:11.545243 754 tablet_service.cc:1468] Processing CreateTablet for tablet 589a42f0f7b94ae88d09d0f7de60e175 (DEFAULT_TABLE table=default.Table_0221d3b2f7554e1a9a8159c7fade3b33 [id=aed15b58d72b49c2b0fddd6d6c2a96f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.546391 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_628c945c37c84b88a1ed70deaadd49a6" } modify_external_catalogs: true
I20250625 13:52:11.546537 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_628c945c37c84b88a1ed70deaadd49a6" } modify_external_catalogs: true
I20250625 13:52:11.548259 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ba34195043ab4dd79dea215f57cbb3cc" } new_table_name: "default.Table_312dbec5d3ab4810aeb370951ebfafde" modify_external_catalogs: true
I20250625 13:52:11.553942 1170 tablet_bootstrap.cc:492] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:11.554085 1170 ts_tablet_manager.cc:1397] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.647s user 0.009s sys 0.004s
I20250625 13:52:11.554805 1170 raft_consensus.cc:357] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.554982 1170 raft_consensus.cc:383] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.555039 1170 raft_consensus.cc:738] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.555612 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f5f3d8e484094d71808796036298fd34" } modify_external_catalogs: true
I20250625 13:52:11.552346 888 tablet_service.cc:1468] Processing CreateTablet for tablet 0bebb74334374efa90898ab97a13d509 (DEFAULT_TABLE table=default.Table_5d17ee811c9c4e2dbfaae512da4b46f6 [id=b429e1ee7cdc42ed9deff035bca5e9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.556025 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bebb74334374efa90898ab97a13d509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.543159 893 tablet_service.cc:1468] Processing CreateTablet for tablet bb993cc5648f44a6bf8c15544fff7cca (DEFAULT_TABLE table=default.Table_ee7a863cc44d4067a1f5a29e6fe267af [id=28eab56011a1491abfc13e26d286d5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.559854 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb993cc5648f44a6bf8c15544fff7cca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.560747 1044 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.007s sys 0.050s
W20250625 13:52:11.560842 1044 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.007s sys 0.050s
I20250625 13:52:11.560896 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c79a2f6544493ba3f3ec20ea67fa95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.563522 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c89a899837f40a8ac9eed9f1d68e420. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.563844 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 529c3bf8e72f46948cfb864445ff51f3 (table default.Table_f3b3e3e07eb84bfa8a9396b3161a9114 [id=5227457618334dd0ad007356f6a79539], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 529c3bf8e72f46948cfb864445ff51f3 consensus state has no leader
I20250625 13:52:11.567538 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1503714b43c54b738a034ca7d0a8a602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.568645 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f1050a29b540558aaf1e31118e4531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.573951 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba762a8edb8847368103e5bede9e9d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.578656 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589a42f0f7b94ae88d09d0f7de60e175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.555052 1171 tablet_bootstrap.cc:492] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:11.552913 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:11.570788 877 tablet_service.cc:1468] Processing CreateTablet for tablet 78c79a2f6544493ba3f3ec20ea67fa95 (DEFAULT_TABLE table=default.Table_e899e0383341441d9eeca197fd80a8be [id=425d7c5e4b0b4fb69c3b61fc1ca233fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.579635 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c79a2f6544493ba3f3ec20ea67fa95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.580832 1171 tablet_bootstrap.cc:654] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.571678 891 tablet_service.cc:1468] Processing CreateTablet for tablet 3a6220e6861d4f8099258fbfabaf7da8 (DEFAULT_TABLE table=default.Table_b75460e88d6f4692a0992d1601f9783d [id=c9395724c7eb4ab59a5e72d7906c624f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.586393 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6220e6861d4f8099258fbfabaf7da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.587469 1166 tablet_bootstrap.cc:700] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent opening tablet: real 0.145s user 0.002s sys 0.000s
I20250625 13:52:11.587831 1166 tablet_bootstrap.cc:654] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:11.591387 1023 thread.cc:640] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.000s sys 0.053s
W20250625 13:52:11.591501 1023 thread.cc:606] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.646s user 0.000s sys 0.053s
I20250625 13:52:11.594043 1009 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 159) took 1662 ms. Trace:
I20250625 13:52:11.594167 1009 rpcz_store.cc:276] 0625 13:52:09.932040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.932097 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:10.775822 (+843725us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x616000145654. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:10.871764 (+ 95942us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x616000123830. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd29add9 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:11.593995 (+722231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1945550080}
I20250625 13:52:11.598126 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 913bc2f716c14de890bf1efdb0611dde
I20250625 13:52:11.599473 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f5f3d8e484094d71808796036298fd34" } modify_external_catalogs: true
I20250625 13:52:11.601102 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2593db823e974f1eb653470d5a23338f
I20250625 13:52:11.601174 885 tablet_service.cc:1468] Processing CreateTablet for tablet ba762a8edb8847368103e5bede9e9d0b (DEFAULT_TABLE table=default.Table_ee5d5978aaf540b8bc77e7c4bbe55258 [id=9c5c587ce78d469ca11f308fd71a6aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.601565 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba762a8edb8847368103e5bede9e9d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.603976 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 913bc2f716c14de890bf1efdb0611dde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.603940 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 2593db823e974f1eb653470d5a23338f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.619170 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c6963b0fa6a4bce977904672ec5ae28" } modify_external_catalogs: true
I20250625 13:52:11.619472 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c6963b0fa6a4bce977904672ec5ae28" } modify_external_catalogs: true
I20250625 13:52:11.620280 1170 consensus_queue.cc:260] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.622519 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7e1bff138de841eda69caac9be5438bb" } modify_external_catalogs: true
I20250625 13:52:11.622710 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7e1bff138de841eda69caac9be5438bb" } modify_external_catalogs: true
W20250625 13:52:11.624002 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 913bc2f716c14de890bf1efdb0611dde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913bc2f716c14de890bf1efdb0611dde already in progress: creating tablet
W20250625 13:52:11.625085 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2593db823e974f1eb653470d5a23338f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2593db823e974f1eb653470d5a23338f already in progress: creating tablet
I20250625 13:52:11.626922 1170 ts_tablet_manager.cc:1428] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.073s user 0.001s sys 0.002s
I20250625 13:52:11.627624 1170 tablet_bootstrap.cc:492] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:11.628257 885 tablet_service.cc:1468] Processing CreateTablet for tablet 589a42f0f7b94ae88d09d0f7de60e175 (DEFAULT_TABLE table=default.Table_0221d3b2f7554e1a9a8159c7fade3b33 [id=aed15b58d72b49c2b0fddd6d6c2a96f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.628806 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589a42f0f7b94ae88d09d0f7de60e175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.633462 877 tablet_service.cc:1468] Processing CreateTablet for tablet 1503714b43c54b738a034ca7d0a8a602 (DEFAULT_TABLE table=default.Table_01f30a93171b4189adaa5b5c283a43e1 [id=13a2c478d08d4f3d80332697879d54d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.636760 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1503714b43c54b738a034ca7d0a8a602. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.639006 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5abd4d83514458c95ec62290b709bc3 consensus state has no leader
I20250625 13:52:11.634938 883 tablet_service.cc:1468] Processing CreateTablet for tablet 6c89a899837f40a8ac9eed9f1d68e420 (DEFAULT_TABLE table=default.Table_a18783dacb0b4bd09053d2559ed93252 [id=6c7502c2b02840f7a7f8c8be17b750d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.639397 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c89a899837f40a8ac9eed9f1d68e420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.640380 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3a157223727844e1be7e975b3edd19fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.641112 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a157223727844e1be7e975b3edd19fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.643136 889 tablet_service.cc:1468] Processing CreateTablet for tablet 19f1050a29b540558aaf1e31118e4531 (DEFAULT_TABLE table=default.Table_76340fa76a5048f9b56482e7e2563551 [id=0ecf1056ada74055bf862e9b3e86ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.643865 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ecdaa626099d4adaa1f78a0ea582bfa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.644265 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f1050a29b540558aaf1e31118e4531. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.644376 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecdaa626099d4adaa1f78a0ea582bfa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.645455 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c9d010744e88406a828c6ba7b76eda3e
I20250625 13:52:11.647027 756 tablet_service.cc:1515] Processing DeleteTablet for tablet c9d010744e88406a828c6ba7b76eda3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:11.647660 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c9d010744e88406a828c6ba7b76eda3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9d010744e88406a828c6ba7b76eda3e already in progress: creating tablet
I20250625 13:52:11.573374 888 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd0d216fb394f31afa4478ad2097fe5 (DEFAULT_TABLE table=default.Table_fdada5e5d21042ecb3d4b84dc1f0b299 [id=9cf4473ffa5e415b872597528f5aea2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.648329 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd0d216fb394f31afa4478ad2097fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.657238 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_786e38e4db2a4cc1a486f0f8bc54da13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.657374 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b61d4f7f0474aac9b46c532f6f403a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:11.657967 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_786e38e4db2a4cc1a486f0f8bc54da13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.660620 1021 tablet_service.cc:1468] Processing CreateTablet for tablet fee74694c02b4fa2a405c57ae7c22a95 (DEFAULT_TABLE table=default.Table_6b8809b44d10450497c7a9f223dda4de [id=a90f9e531c534670917422e193674d1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.661391 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 4692e7df2014475c86de7ed7735ef3c0 (DEFAULT_TABLE table=default.Table_0b55a0fa335e4435a1f8c85d61d7c13b [id=4a377b59e94f408c871c5a1839d18e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.662129 595 master_service.cc:948] ReplaceTablet: received request to replace tablet bd4cc91d274a45a393beffaaa59a35f0 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:11.664593 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9222dfc69aed449eb67eb228923b064f" } new_table_name: "default.Table_7802866074204a24ba0302a41c6bbaf7" modify_external_catalogs: true
I20250625 13:52:11.666556 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:11.666949 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4293c9800b3c40498bd0d764bf20c537"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.667567 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4293c9800b3c40498bd0d764bf20c537 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.667793 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_279ed5effac34385a5eb2ea2439ec1ae" } new_table_name: "default.Table_cd806d72367a416395dfb798a9ba8084" modify_external_catalogs: true
I20250625 13:52:11.668833 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_338af88ef39143f0823658cf2b61c83b" } new_table_name: "default.Table_958988b40e5e4c2db7d10efe4a78a7bc" modify_external_catalogs: true
I20250625 13:52:11.669334 1170 tablet_bootstrap.cc:654] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.669777 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_163659f4838a45f698a8407f2ec827f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.670240 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_163659f4838a45f698a8407f2ec827f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.670284 2195 raft_consensus.cc:491] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:11.670413 985 consensus_peers.cc:487] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be3313ec5dfb41b68662d3ff4c8c94f9. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:11.670447 2195 raft_consensus.cc:513] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.671007 2195 leader_election.cc:290] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:11.671666 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:11.672154 711 leader_election.cc:343] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.672526 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee74694c02b4fa2a405c57ae7c22a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.672984 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bd3d18582db04d18829da98f82d6ee0c" } new_table_name: "default.Table_01e8a56cd3c748c2bf70572909dd7215" modify_external_catalogs: true
I20250625 13:52:11.675472 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4692e7df2014475c86de7ed7735ef3c0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.675606 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a1a9a69d26c14a62b26e0e90157782af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a9a69d26c14a62b26e0e90157782af already in progress: creating tablet
I20250625 13:52:11.677183 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6391ab0c8b3544d0b92e70aa13402edd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.677676 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6391ab0c8b3544d0b92e70aa13402edd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.679620 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:11.680202 711 leader_election.cc:343] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.680323 711 leader_election.cc:304] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:11.680857 2195 raft_consensus.cc:2747] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.683890 2267 consensus_queue.cc:1035] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:11.685257 895 tablet_service.cc:1468] Processing CreateTablet for tablet 05f205733f3d4c81b9e71709c7559660 (DEFAULT_TABLE table=default.Table_ca0d7155da664e88b759e29a7c5d554b [id=6ad809b2835543f7987c9d35b948a21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.685812 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f205733f3d4c81b9e71709c7559660. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.692250 1166 tablet_bootstrap.cc:492] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:11.692493 1166 ts_tablet_manager.cc:1397] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.251s user 0.004s sys 0.000s
W20250625 13:52:11.692742 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e29bade90b184d2483cb1fe85033a03c consensus state has no leader
W20250625 13:52:11.695619 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f177c0da5ab41a784a512d6eeb48d29 consensus state has no leader
W20250625 13:52:11.696298 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1068b65734bc4c29ba8b09a7a3d575b1 consensus state has no leader
I20250625 13:52:11.697897 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f5e71dc7423c41da9b00ff656cb53b90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.698537 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5e71dc7423c41da9b00ff656cb53b90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.699589 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_41e5ba0098654109a19569d17b1ab1b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.700155 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41e5ba0098654109a19569d17b1ab1b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.703703 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 4acfcd35d6434ee59b9b0d831a59938c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.704224 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 05f205733f3d4c81b9e71709c7559660 (DEFAULT_TABLE table=default.Table_ca0d7155da664e88b759e29a7c5d554b [id=6ad809b2835543f7987c9d35b948a21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.704666 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f205733f3d4c81b9e71709c7559660. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.708807 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4111b95923ed4d73b7eca60b964d9494 consensus state has no leader
I20250625 13:52:11.710511 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ed99b177ce51470a970fe7a1d9e6e25e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.710966 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd0d216fb394f31afa4478ad2097fe5 (DEFAULT_TABLE table=default.Table_fdada5e5d21042ecb3d4b84dc1f0b299 [id=9cf4473ffa5e415b872597528f5aea2a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:11.711161 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed99b177ce51470a970fe7a1d9e6e25e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.711306 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd0d216fb394f31afa4478ad2097fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.713979 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.714500 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.715956 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 3a6220e6861d4f8099258fbfabaf7da8 (DEFAULT_TABLE table=default.Table_b75460e88d6f4692a0992d1601f9783d [id=c9395724c7eb4ab59a5e72d7906c624f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.716454 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a6220e6861d4f8099258fbfabaf7da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.719646 2267 consensus_queue.cc:1035] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:11.720991 1166 raft_consensus.cc:357] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.721172 1166 raft_consensus.cc:383] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.721230 1166 raft_consensus.cc:738] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.685956 1171 tablet_bootstrap.cc:492] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.721623 1171 ts_tablet_manager.cc:1397] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.167s user 0.017s sys 0.025s
I20250625 13:52:11.722373 1171 raft_consensus.cc:357] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.722568 1171 raft_consensus.cc:383] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.722645 1171 raft_consensus.cc:738] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.722862 1171 consensus_queue.cc:260] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.723407 1171 ts_tablet_manager.cc:1428] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:11.723673 1171 tablet_bootstrap.cc:492] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:11.725396 1171 tablet_bootstrap.cc:654] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.729157 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 78c79a2f6544493ba3f3ec20ea67fa95 (DEFAULT_TABLE table=default.Table_e899e0383341441d9eeca197fd80a8be [id=425d7c5e4b0b4fb69c3b61fc1ca233fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.729584 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c79a2f6544493ba3f3ec20ea67fa95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.735713 595 catalog_manager.cc:5582] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:11.735927 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6c89a899837f40a8ac9eed9f1d68e420 (DEFAULT_TABLE table=default.Table_a18783dacb0b4bd09053d2559ed93252 [id=6c7502c2b02840f7a7f8c8be17b750d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.736126 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:11.736366 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c89a899837f40a8ac9eed9f1d68e420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.743912 756 tablet_service.cc:1468] Processing CreateTablet for tablet 285912ccbf5a4e5bab53c48d1c024bb8 (DEFAULT_TABLE table=default.Table_0977431f341c4a25825dd7be66a01913 [id=ffb19ea26cba49fa999217f71a4132d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.744299 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 285912ccbf5a4e5bab53c48d1c024bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.744334 1166 consensus_queue.cc:260] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.744808 754 tablet_service.cc:1468] Processing CreateTablet for tablet a33c304d652e4e61a3bcfb3ab7067244 (DEFAULT_TABLE table=default.Table_2a80a6c4ae994eda8c3f3fb7b126616c [id=03b41b1e711a4af484ad35073f456efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.745189 742 tablet_service.cc:1468] Processing CreateTablet for tablet 234be8f208324ddd969555a2e88e2546 (DEFAULT_TABLE table=default.Table_521bb02a0fa64038b747db4f73fa8c90 [id=d1f78b212b5b4608b3bd1edcd80ed747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.745342 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33c304d652e4e61a3bcfb3ab7067244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.745618 750 tablet_service.cc:1468] Processing CreateTablet for tablet 80025a9dbde648a29a9a75ca91088634 (DEFAULT_TABLE table=default.Table_7c5253f2510845c58dc974a5a0840b62 [id=37fda201e6e54220873b8631039f6765]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.746461 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80025a9dbde648a29a9a75ca91088634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.747045 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234be8f208324ddd969555a2e88e2546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.747550 744 tablet_service.cc:1468] Processing CreateTablet for tablet a2151ff900b042399199081137c31fd0 (DEFAULT_TABLE table=default.Table_7a3b5892cacf49f29167f2125d14f34a [id=3a329f34821443e6beb7b59680575dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.747838 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2151ff900b042399199081137c31fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.751544 747 tablet_service.cc:1468] Processing CreateTablet for tablet d6ce665e69c64e45bac0bdef14f57c3b (DEFAULT_TABLE table=default.Table_3f19c3d36661484dac5b43b45c74851c [id=07bf9483068c420f8be59b01b93e5bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.751891 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2191b836bb3044c89fee13c869d9e07d (DEFAULT_TABLE table=default.Table_afac01f597474109bbc2f29c5279b1fa [id=60d58942d03b4b04be7f96c08f758a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.752492 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ce665e69c64e45bac0bdef14f57c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.757465 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b61d4f7f0474aac9b46c532f6f403a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:11.757912 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 97272adfbe254967b099a597ca60d464 (DEFAULT_TABLE table=default.Table_220446a0ea6a427d9940bdb6fd32dad7 [id=ddde40591af64880ad0321b8b180f7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.758150 1166 ts_tablet_manager.cc:1428] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20250625 13:52:11.758415 1166 tablet_bootstrap.cc:492] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:11.758854 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2191b836bb3044c89fee13c869d9e07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.759780 1166 tablet_bootstrap.cc:654] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.766458 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.769285 1165 tablet_replica.cc:331] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:11.769573 1165 raft_consensus.cc:2241] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.769748 1165 raft_consensus.cc:2270] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:11.772310 1165 ts_tablet_manager.cc:1905] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:11.766350 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 589a42f0f7b94ae88d09d0f7de60e175 (DEFAULT_TABLE table=default.Table_0221d3b2f7554e1a9a8159c7fade3b33 [id=aed15b58d72b49c2b0fddd6d6c2a96f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.776726 2204 raft_consensus.cc:491] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.773080 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 19f1050a29b540558aaf1e31118e4531 (DEFAULT_TABLE table=default.Table_76340fa76a5048f9b56482e7e2563551 [id=0ecf1056ada74055bf862e9b3e86ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.776993 2204 raft_consensus.cc:513] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:11.771492 985 consensus_peers.cc:487] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 063a793e0d7747668d2728fdc2160b5e. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:11.758240 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97272adfbe254967b099a597ca60d464. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.777793 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589a42f0f7b94ae88d09d0f7de60e175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.783172 1165 ts_tablet_manager.cc:1918] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:11.783340 1165 log.cc:1199] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/68b8c8aa8036454298bf1e81bab91c01
I20250625 13:52:11.783666 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f1050a29b540558aaf1e31118e4531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.783689 1165 ts_tablet_manager.cc:1939] T 68b8c8aa8036454298bf1e81bab91c01 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:11.784900 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 68b8c8aa8036454298bf1e81bab91c01 (table default.Table_068bedd508da462681c0138ddd8964e3 [id=3a4d573b5eb84c639bf289ec308eb50b]) successfully deleted
I20250625 13:52:11.785239 1027 tablet_service.cc:1468] Processing CreateTablet for tablet ba762a8edb8847368103e5bede9e9d0b (DEFAULT_TABLE table=default.Table_ee5d5978aaf540b8bc77e7c4bbe55258 [id=9c5c587ce78d469ca11f308fd71a6aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.785648 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba762a8edb8847368103e5bede9e9d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.787871 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604
I20250625 13:52:11.788600 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 54ea4c4d7756433d85fe9b816b4816ae
I20250625 13:52:11.789431 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7
I20250625 13:52:11.789835 589 catalog_manager.cc:6652] ReplaceTablet: tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 deleted and replaced by tablet d60228a7b8b7496b89733753ec0b88d3
I20250625 13:52:11.790899 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 with delete_type TABLET_DATA_DELETED (replaced by tablet d60228a7b8b7496b89733753ec0b88d3) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.790900 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:11.675848 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 973c483c87b84ea09b6c7d556e54b490 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 973c483c87b84ea09b6c7d556e54b490 already in progress: creating tablet
W20250625 13:52:11.791517 581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 115642us
W20250625 13:52:11.791759 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e3939ef22ea344328721dfba67d933d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3939ef22ea344328721dfba67d933d1 already in progress: creating tablet
W20250625 13:52:11.792021 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bd4cc91d274a45a393beffaaa59a35f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4cc91d274a45a393beffaaa59a35f0 already in progress: creating tablet
W20250625 13:52:11.792243 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 47c70919181d4c5392f65df221ec6348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c70919181d4c5392f65df221ec6348 already in progress: creating tablet
W20250625 13:52:11.792459 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 already in progress: creating tablet
I20250625 13:52:11.792424 744 tablet_service.cc:1468] Processing CreateTablet for tablet bbd3200bd1384187978aeb7089ec4e03 (DEFAULT_TABLE table=default.Table_4215c08737a746b59dd7eb42a6a34ba5 [id=ef0b6416799545eba962b9bf74f5256f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:11.792840 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c9ae350a9dbf476081dc56a709de42c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9ae350a9dbf476081dc56a709de42c8 already in progress: creating tablet
W20250625 13:52:11.793093 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2770194711ec4a86bb1354517b095f60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2770194711ec4a86bb1354517b095f60 already in progress: creating tablet
I20250625 13:52:11.793125 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd3200bd1384187978aeb7089ec4e03. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:11.793308 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 38c954e01db443fc802d76bbbb853c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38c954e01db443fc802d76bbbb853c14 already in progress: creating tablet
W20250625 13:52:11.793529 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 422f4d02482942dba453ff560890137b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 422f4d02482942dba453ff560890137b already in progress: creating tablet
W20250625 13:52:11.793743 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a8334d8187d2473582f0057193874b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8334d8187d2473582f0057193874b74 already in progress: creating tablet
W20250625 13:52:11.793972 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7507d50e9f464043b8901491c57391d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7507d50e9f464043b8901491c57391d6 already in progress: creating tablet
W20250625 13:52:11.794174 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 723246e1e6824faf9f9d39c3ea941fe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 723246e1e6824faf9f9d39c3ea941fe4 already in progress: creating tablet
W20250625 13:52:11.794389 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet af704cdf44ef42f7b5ea10b199d0f017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af704cdf44ef42f7b5ea10b199d0f017 already in progress: creating tablet
W20250625 13:52:11.794631 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5d2ce669ecd441939be8e8c71eea7bfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2ce669ecd441939be8e8c71eea7bfa already in progress: creating tablet
I20250625 13:52:11.798969 2204 leader_election.cc:290] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:11.799523 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:11.800048 849 leader_election.cc:343] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.800299 749 tablet_service.cc:1468] Processing CreateTablet for tablet 82b8afc20aa446e3a714000954f99b0e (DEFAULT_TABLE table=default.Table_d5806194428c4cb9927a09061a2d6533 [id=2c3f0413818e41358b7952e366951cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.800662 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b8afc20aa446e3a714000954f99b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.698722 1168 ts_tablet_manager.cc:1918] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:11.801028 1168 log.cc:1199] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/8fdca99b734448118e0c42227c78e58f
I20250625 13:52:11.801156 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:11.801323 774 raft_consensus.cc:2466] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:11.801402 1168 ts_tablet_manager.cc:1939] T 8fdca99b734448118e0c42227c78e58f P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:11.801731 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 54ea4c4d7756433d85fe9b816b4816ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.805323 1170 tablet_bootstrap.cc:492] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:11.805373 2204 raft_consensus.cc:491] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.805521 2204 raft_consensus.cc:513] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.806504 2204 leader_election.cc:290] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:11.817109 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6024520893584b229315b8f2b59329e0 (DEFAULT_TABLE table=default.Table_943353535aac4d8085a467fcadb1af36 [id=a2891068fff94edb8e9bdd7614002107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.817514 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6024520893584b229315b8f2b59329e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.818585 1170 ts_tablet_manager.cc:1397] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.191s user 0.000s sys 0.004s
I20250625 13:52:11.806856 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38a23b08a03b48eaa051f670eb7dac65" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:11.819386 849 leader_election.cc:343] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.819769 1170 raft_consensus.cc:357] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.820024 1170 raft_consensus.cc:383] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.820166 1170 raft_consensus.cc:738] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.820547 1170 consensus_queue.cc:260] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.824832 1170 ts_tablet_manager.cc:1428] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:52:11.825351 1170 tablet_bootstrap.cc:492] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:11.830652 851 leader_election.cc:304] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
W20250625 13:52:11.831148 851 leader_election.cc:343] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 38a23b08a03b48eaa051f670eb7dac65
I20250625 13:52:11.831344 851 leader_election.cc:304] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:11.831502 2204 raft_consensus.cc:2802] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:11.831653 2204 raft_consensus.cc:491] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:11.831765 2204 raft_consensus.cc:3058] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:11.832953 2192 raft_consensus.cc:2747] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.836503 2204 raft_consensus.cc:513] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.773998 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 1503714b43c54b738a034ca7d0a8a602 (DEFAULT_TABLE table=default.Table_01f30a93171b4189adaa5b5c283a43e1 [id=13a2c478d08d4f3d80332697879d54d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.837338 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1503714b43c54b738a034ca7d0a8a602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.838868 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:11.839107 1171 tablet_bootstrap.cc:492] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.839228 1171 ts_tablet_manager.cc:1397] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.001s
W20250625 13:52:11.839664 849 leader_election.cc:343] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.839694 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0bebb74334374efa90898ab97a13d509 (DEFAULT_TABLE table=default.Table_5d17ee811c9c4e2dbfaae512da4b46f6 [id=b429e1ee7cdc42ed9deff035bca5e9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:11.839879 1171 raft_consensus.cc:357] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:11.840013 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bebb74334374efa90898ab97a13d509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.840030 1171 raft_consensus.cc:383] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.840087 1171 raft_consensus.cc:738] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.841193 1171 consensus_queue.cc:260] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:11.841686 1171 ts_tablet_manager.cc:1428] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:52:11.841951 2204 leader_election.cc:290] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:11.842429 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 793) took 3719 ms. Trace:
I20250625 13:52:11.842552 1168 rpcz_store.cc:276] 0625 13:52:08.122730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.123178 (+ 448us) service_pool.cc:225] Handling call
0625 13:52:11.268192 (+3145014us) ts_tablet_manager.cc:1145] Deleting tablet 8fdca99b734448118e0c42227c78e58f
0625 13:52:11.268205 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.659886 (+391681us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:11.698624 (+ 38738us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:11.842412 (+143788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3144909}
I20250625 13:52:11.842868 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:11.843043 774 raft_consensus.cc:3058] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:11.843092 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 797) took 3622 ms. Trace:
I20250625 13:52:11.843161 1168 rpcz_store.cc:276] 0625 13:52:08.220346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.220400 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:11.842948 (+3622548us) ts_tablet_manager.cc:1145] Deleting tablet 3b8e232f80724e38992aa5d66fd0caf3
0625 13:52:11.842959 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.843069 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3622371}
I20250625 13:52:11.845681 774 raft_consensus.cc:2466] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:11.847144 1007 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 160) took 1108 ms. Trace:
I20250625 13:52:11.847366 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_997801fa9593458d9642268646093de6" } modify_external_catalogs: true
I20250625 13:52:11.847352 1007 rpcz_store.cc:276] 0625 13:52:10.738754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.739049 (+ 295us) service_pool.cc:225] Handling call
0625 13:52:11.096891 (+357842us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x60c0001658f8. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fb9390ab 00007f18fb8f6353 00007f18fd29bf08 00007f18fd29b3ca 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95
0625 13:52:11.563453 (+466562us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x616000220954. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:11.847095 (+283642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1031366784}
I20250625 13:52:11.847819 851 leader_election.cc:304] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:11.848270 2210 raft_consensus.cc:2802] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:11.848399 2210 raft_consensus.cc:695] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:11.848634 2210 consensus_queue.cc:237] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.852588 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8be17994b5324638bcceef7134adbaf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.857224 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e64d8f11a6ed4105a26c31a57a80f3aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.857411 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 799) took 3619 ms. Trace:
I20250625 13:52:11.857488 1168 rpcz_store.cc:276] 0625 13:52:08.237798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.237872 (+ 74us) service_pool.cc:225] Handling call
0625 13:52:11.857282 (+3619410us) ts_tablet_manager.cc:1145] Deleting tablet 22bcbfcfeb0a439288622994b2d245b2
0625 13:52:11.857292 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.857390 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3619255}
I20250625 13:52:11.857800 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 800) took 3565 ms. Trace:
I20250625 13:52:11.857913 1168 rpcz_store.cc:276] 0625 13:52:08.292380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.294915 (+ 2535us) service_pool.cc:225] Handling call
0625 13:52:11.857703 (+3562788us) ts_tablet_manager.cc:1145] Deleting tablet 8cc353b88a124fac801b8898b5a5f72c
0625 13:52:11.857711 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.857786 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3562424}
I20250625 13:52:11.858471 1168 tablet_replica.cc:331] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:11.858700 1168 raft_consensus.cc:2241] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.858947 1168 raft_consensus.cc:2270] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:11.859925 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 already in progress: creating tablet
I20250625 13:52:11.864956 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:11.869781 1171 tablet_bootstrap.cc:492] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:11.871270 1171 tablet_bootstrap.cc:654] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.871776 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ecf99b86cf4fe1bfac2b0907ef8662. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:11.872727 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_734235bd5dc045678fdad3d130384a52" } modify_external_catalogs: true
I20250625 13:52:11.872997 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_734235bd5dc045678fdad3d130384a52" } modify_external_catalogs: true
I20250625 13:52:11.874483 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0977431f341c4a25825dd7be66a01913" } modify_external_catalogs: true
I20250625 13:52:11.874697 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0977431f341c4a25825dd7be66a01913" } modify_external_catalogs: true
I20250625 13:52:11.875131 592 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet dad4ea8cf44944b08f53428c95809daf (table default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f]). Expected version 3 got 0
I20250625 13:52:11.875475 592 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 6e529f5bf13f4ee69e66402b1ee252d4 (table default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716]). Expected version 2 got 0
I20250625 13:52:11.878494 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_034b191549844f928bcd64e6d87483dd" } new_table_name: "default.Table_40e114808bdd4477ae9ba7f8f92f6f8c" modify_external_catalogs: true
I20250625 13:52:11.879298 1170 tablet_bootstrap.cc:654] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.879716 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_997801fa9593458d9642268646093de6" } modify_external_catalogs: true
I20250625 13:52:11.879798 1168 ts_tablet_manager.cc:1905] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:11.880839 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8be17994b5324638bcceef7134adbaf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:11.881498 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e64d8f11a6ed4105a26c31a57a80f3aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:11.881533 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 already in progress: creating tablet
W20250625 13:52:11.882048 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 54ea4c4d7756433d85fe9b816b4816ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54ea4c4d7756433d85fe9b816b4816ae already in progress: creating tablet
I20250625 13:52:11.883164 591 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
W20250625 13:52:11.885669 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3c75deb64c74f0282efeed827659ec2 consensus state has no leader
I20250625 13:52:11.887010 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9302ac10a29c4b0c82268eb767615989
W20250625 13:52:11.872730 581 catalog_manager.cc:4895] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 12a7db105cb34bb2ac7a411256c43cae because the tablet was not found. No further retry: Not found: Tablet not found: 12a7db105cb34bb2ac7a411256c43cae
W20250625 13:52:11.890487 581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 95645.1us
I20250625 13:52:11.891053 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bdc5395c1de349b9b20d28625b8751f7
W20250625 13:52:11.892151 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4acfcd35d6434ee59b9b0d831a59938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4acfcd35d6434ee59b9b0d831a59938c already in progress: creating tablet
I20250625 13:52:11.892295 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 9302ac10a29c4b0c82268eb767615989 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:11.892797 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4b61d4f7f0474aac9b46c532f6f403a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b61d4f7f0474aac9b46c532f6f403a0 already in progress: creating tablet
I20250625 13:52:11.895793 1166 tablet_bootstrap.cc:492] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:11.896004 1166 ts_tablet_manager.cc:1397] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.138s user 0.002s sys 0.001s
I20250625 13:52:11.897039 1166 raft_consensus.cc:357] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.897329 1166 raft_consensus.cc:383] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.897433 1166 raft_consensus.cc:738] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.897745 1166 consensus_queue.cc:260] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.898319 1166 ts_tablet_manager.cc:1428] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:11.901464 1166 tablet_bootstrap.cc:492] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:11.903191 1166 tablet_bootstrap.cc:654] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.904727 749 tablet_service.cc:1515] Processing DeleteTablet for tablet bdc5395c1de349b9b20d28625b8751f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.912339 1168 ts_tablet_manager.cc:1918] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:11.912597 1047 raft_consensus.cc:3058] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:11.934506 1168 log.cc:1199] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/063a793e0d7747668d2728fdc2160b5e
I20250625 13:52:11.935410 1170 tablet_bootstrap.cc:492] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:11.935575 1168 ts_tablet_manager.cc:1939] T 063a793e0d7747668d2728fdc2160b5e P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:11.935626 1170 ts_tablet_manager.cc:1397] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.001s
I20250625 13:52:11.936353 1170 raft_consensus.cc:357] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.936632 1170 raft_consensus.cc:383] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.936822 1170 raft_consensus.cc:738] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.937239 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 802) took 3626 ms. Trace:
I20250625 13:52:11.937337 1168 rpcz_store.cc:276] 0625 13:52:08.311195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.311245 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:11.858433 (+3547188us) ts_tablet_manager.cc:1145] Deleting tablet 063a793e0d7747668d2728fdc2160b5e
0625 13:52:11.858441 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.895992 (+ 37551us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:11.912235 (+ 16243us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:11.937222 (+ 24987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3546980}
I20250625 13:52:11.937273 1170 consensus_queue.cc:260] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.929929 2210 raft_consensus.cc:491] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:11.937893 2210 raft_consensus.cc:513] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.938757 2210 leader_election.cc:290] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:11.938887 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "024f6cb363594420b0d5480fa79d156b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:11.939451 849 leader_election.cc:343] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:11.939947 1170 ts_tablet_manager.cc:1428] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:11.941313 1170 tablet_bootstrap.cc:492] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:11.943763 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 803) took 3625 ms. Trace:
I20250625 13:52:11.944957 1168 rpcz_store.cc:276] 0625 13:52:08.318719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.318777 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:11.943641 (+3624864us) ts_tablet_manager.cc:1145] Deleting tablet 066db01f791542a38cfb6e408c4836ed
0625 13:52:11.943655 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.943747 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3624706}
I20250625 13:52:11.945528 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 804) took 3622 ms. Trace:
I20250625 13:52:11.946592 1166 tablet_bootstrap.cc:492] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:11.946686 1168 rpcz_store.cc:276] 0625 13:52:08.322997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.323049 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:11.945422 (+3622373us) ts_tablet_manager.cc:1145] Deleting tablet da6bcb2055b94749879607559e62f6eb
0625 13:52:11.945433 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.945513 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3622185}
I20250625 13:52:11.946848 1166 ts_tablet_manager.cc:1397] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250625 13:52:11.947703 1166 raft_consensus.cc:357] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.947986 1166 raft_consensus.cc:383] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.948171 1166 raft_consensus.cc:738] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.948684 1166 consensus_queue.cc:260] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.949026 1168 tablet_replica.cc:331] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:11.949250 1168 raft_consensus.cc:2241] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.950031 1168 raft_consensus.cc:2270] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:11.951336 1168 ts_tablet_manager.cc:1905] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:11.961051 1166 ts_tablet_manager.cc:1428] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250625 13:52:11.966292 851 leader_election.cc:343] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 024f6cb363594420b0d5480fa79d156b
I20250625 13:52:11.966567 851 leader_election.cc:304] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:11.966885 2210 raft_consensus.cc:2747] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:11.975201 1170 tablet_bootstrap.cc:654] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.975843 1166 tablet_bootstrap.cc:492] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:11.979667 1166 tablet_bootstrap.cc:654] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:11.987977 1171 tablet_bootstrap.cc:492] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:11.988127 1171 ts_tablet_manager.cc:1397] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.118s user 0.002s sys 0.003s
I20250625 13:52:11.988960 1171 raft_consensus.cc:357] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.989154 1171 raft_consensus.cc:383] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:11.989231 1171 raft_consensus.cc:738] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:11.989485 1171 consensus_queue.cc:260] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:11.990000 1168 ts_tablet_manager.cc:1918] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:11.991065 581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 98259us
W20250625 13:52:11.894814 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9302ac10a29c4b0c82268eb767615989 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9302ac10a29c4b0c82268eb767615989 already in progress: creating tablet
I20250625 13:52:11.915328 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1f8b426bfe4f426a9840a85d92052857" } new_table_name: "default.Table_a7af3ed3af9f441c85af4605424c5f90" modify_external_catalogs: true
W20250625 13:52:11.992589 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 97785.7us
W20250625 13:52:11.992986 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bdc5395c1de349b9b20d28625b8751f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdc5395c1de349b9b20d28625b8751f7 already in progress: creating tablet
I20250625 13:52:11.917122 590 catalog_manager.cc:5582] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:11.993840 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
I20250625 13:52:11.994082 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (table default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916]). Expected version 5 got 0
I20250625 13:52:11.917601 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a3e83f37adb44078818e834e91f4e340" } new_table_name: "default.Table_3d4060e3d41e48a788546c077cc9c7aa" modify_external_catalogs: true
I20250625 13:52:11.917914 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a61060462ad540c8b2315b33af732d56" } new_table_name: "default.Table_e3a91abc63c1411da5c7456836ac5862" modify_external_catalogs: true
I20250625 13:52:11.937893 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2a0fc7dfe9884d9eb17952b2d9d3d5ee" } new_table_name: "default.Table_192bf6df15cb499a89d7a61c09717d78" modify_external_catalogs: true
I20250625 13:52:11.996376 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 8063122ea7b24b6790f5c3cc884e09be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:11.996762 1165 tablet_replica.cc:331] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:11.996971 1165 raft_consensus.cc:2241] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:11.939541 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b7652d48c11040f19386af4a01ea6be6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:11.997136 1165 raft_consensus.cc:2270] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:11.997458 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7652d48c11040f19386af4a01ea6be6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.997889 1165 ts_tablet_manager.cc:1905] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:11.940623 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0566054eac7f44f0801ea96f33d099fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.998620 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0566054eac7f44f0801ea96f33d099fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.941082 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3e4c58462d2844a6be928491924b59fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:11.999732 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e4c58462d2844a6be928491924b59fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:11.993677 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 8fdca99b734448118e0c42227c78e58f (table default.Table_cd4b8059082844b99439152f1be88d22 [id=d5037bf1fc844a148103da5d2ae95193]) successfully deleted
I20250625 13:52:12.004961 2266 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 13:52:12.005239 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3b8e232f80724e38992aa5d66fd0caf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b8e232f80724e38992aa5d66fd0caf3 already in progress: creating tablet
W20250625 13:52:12.005506 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 22bcbfcfeb0a439288622994b2d245b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bcbfcfeb0a439288622994b2d245b2 already in progress: creating tablet
W20250625 13:52:12.005733 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8cc353b88a124fac801b8898b5a5f72c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cc353b88a124fac801b8898b5a5f72c already in progress: creating tablet
I20250625 13:52:12.006358 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 063a793e0d7747668d2728fdc2160b5e (table default.Table_9c6d8799477b4081b2652a4df3b13762 [id=3398f6ade26340dcae5b0da2eb58034c]) successfully deleted
W20250625 13:52:12.008396 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93003ef3621a4f33acbc7eff54a796e3 consensus state has no leader
I20250625 13:52:12.013500 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_efee3de551ce4c8c96c15ea79ad7fc77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.014721 1168 log.cc:1199] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/5cf73b4ab99c466abcfedf1b200e743b
I20250625 13:52:12.015070 1168 ts_tablet_manager.cc:1939] T 5cf73b4ab99c466abcfedf1b200e743b P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:12.015399 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 805) took 3688 ms. Trace:
I20250625 13:52:12.015486 1168 rpcz_store.cc:276] 0625 13:52:08.326575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.326618 (+ 43us) service_pool.cc:225] Handling call
0625 13:52:11.948972 (+3622354us) ts_tablet_manager.cc:1145] Deleting tablet 5cf73b4ab99c466abcfedf1b200e743b
0625 13:52:11.948985 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:11.962561 (+ 13576us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:11.989892 (+ 27331us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:12.015366 (+ 25474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3622195}
I20250625 13:52:11.990852 1171 ts_tablet_manager.cc:1428] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:11.994288 2271 raft_consensus.cc:491] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.016237 2271 raft_consensus.cc:513] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.016500 1171 tablet_bootstrap.cc:492] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:12.016881 2271 leader_election.cc:290] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:12.018043 1171 tablet_bootstrap.cc:654] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.011801 589 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_39575a8dc48640dbae773650f2417b76 [id=71729b0c8f7046bb8e80bb96f9f2ba71]). Expected version 2 got 0
W20250625 13:52:12.012437 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25c71f11222e48818ea94984a985d6cb consensus state has no leader
I20250625 13:52:12.019619 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e with delete_type TABLET_DATA_DELETED (replaced by tablet 066db01f791542a38cfb6e408c4836ed) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:12.019878 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f4d6be016284dc796945270178b9ee1 consensus state has no leader
I20250625 13:52:12.020166 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d47cfb3324a443628070fd4437024a71" } modify_external_catalogs: true
I20250625 13:52:12.022188 1166 tablet_bootstrap.cc:492] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:12.022363 1166 ts_tablet_manager.cc:1397] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.001s
I20250625 13:52:12.022801 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "067272869c5547648c90f83a34a036aa" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:12.022977 1047 raft_consensus.cc:2466] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:12.023192 1166 raft_consensus.cc:357] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.023397 849 leader_election.cc:304] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:12.023437 1166 raft_consensus.cc:383] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.023520 1166 raft_consensus.cc:738] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.023682 2271 raft_consensus.cc:2802] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:12.023782 2271 raft_consensus.cc:491] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:12.023878 2271 raft_consensus.cc:3058] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:12.024111 851 leader_election.cc:343] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 067272869c5547648c90f83a34a036aa
I20250625 13:52:12.024025 1166 consensus_queue.cc:260] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.024650 1166 ts_tablet_manager.cc:1428] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:12.024981 1166 tablet_bootstrap.cc:492] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:12.026460 2271 raft_consensus.cc:513] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.026597 1166 tablet_bootstrap.cc:654] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.027069 2271 leader_election.cc:290] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:12.028091 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "067272869c5547648c90f83a34a036aa" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:12.028266 1044 raft_consensus.cc:3058] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:12.030797 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 806) took 3694 ms. Trace:
I20250625 13:52:12.030895 1044 raft_consensus.cc:2466] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:12.030916 1168 rpcz_store.cc:276] 0625 13:52:08.336236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.336285 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:12.030641 (+3694356us) ts_tablet_manager.cc:1145] Deleting tablet c0f8e222d6aa4f5582688f874ce04f4f
0625 13:52:12.030653 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:12.030753 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3694194}
I20250625 13:52:12.031447 849 leader_election.cc:304] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:12.032123 2271 raft_consensus.cc:2802] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:12.032313 2271 raft_consensus.cc:695] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:12.032595 2271 consensus_queue.cc:237] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:12.034399 851 leader_election.cc:343] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 067272869c5547648c90f83a34a036aa
I20250625 13:52:12.036227 1165 ts_tablet_manager.cc:1918] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:12.036415 1165 log.cc:1199] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/8063122ea7b24b6790f5c3cc884e09be
I20250625 13:52:12.036826 1165 ts_tablet_manager.cc:1939] T 8063122ea7b24b6790f5c3cc884e09be P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:12.037750 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d47cfb3324a443628070fd4437024a71" } modify_external_catalogs: true
W20250625 13:52:12.038178 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4616015b587a459abfb5403b690d23cf consensus state has no leader
W20250625 13:52:12.038631 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efee3de551ce4c8c96c15ea79ad7fc77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.039857 1168 tablet_replica.cc:331] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:12.040133 1168 raft_consensus.cc:2241] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.040376 1165 tablet_replica.cc:331] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:12.040988 1168 raft_consensus.cc:2270] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.041023 1165 raft_consensus.cc:2241] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.041827 1165 raft_consensus.cc:2270] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.044382 1165 ts_tablet_manager.cc:1905] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.050699 1170 tablet_bootstrap.cc:492] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:12.050920 1170 ts_tablet_manager.cc:1397] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.004s
I20250625 13:52:12.051427 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 8063122ea7b24b6790f5c3cc884e09be (table default.Table_1ffafeb50d654290963bf41dfeb57f55 [id=75f0bde5d60641f8afe279bd82d16ae1]) successfully deleted
I20250625 13:52:12.052150 589 master_service.cc:948] ReplaceTablet: received request to replace tablet c0f461a7a7ef45009279716a3024ee26 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:12.052223 1170 raft_consensus.cc:357] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.052423 1170 raft_consensus.cc:383] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.052477 1170 raft_consensus.cc:738] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.052587 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d3ebd3c017504031acb0eb2ffa156cdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.052695 1170 consensus_queue.cc:260] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.053853 1170 ts_tablet_manager.cc:1428] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:12.051435 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8190e518ba344ab9b9081933923dae13" } modify_external_catalogs: true
W20250625 13:52:12.055328 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3ebd3c017504031acb0eb2ffa156cdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.056017 1170 tablet_bootstrap.cc:492] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:12.056463 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6fa536f6af5e444abbb620755a21d5c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.057646 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fa536f6af5e444abbb620755a21d5c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.058596 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8190e518ba344ab9b9081933923dae13" } modify_external_catalogs: true
I20250625 13:52:12.058996 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f62269079e6d41649f9d7b4c42429ea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.060599 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 066db01f791542a38cfb6e408c4836ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 066db01f791542a38cfb6e408c4836ed already in progress: creating tablet
W20250625 13:52:12.060883 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet da6bcb2055b94749879607559e62f6eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da6bcb2055b94749879607559e62f6eb already in progress: creating tablet
I20250625 13:52:12.060220 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_847080232892457e98242244a03c75f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.063340 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 5cf73b4ab99c466abcfedf1b200e743b (table default.Table_47598e6b1611413784a8c9ab9d1c0a67 [id=8c0cf5af3aee43e9b7676c1be4842093]) successfully deleted
W20250625 13:52:12.063563 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c0f8e222d6aa4f5582688f874ce04f4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f8e222d6aa4f5582688f874ce04f4f already in progress: creating tablet
I20250625 13:52:12.064327 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 913bc2f716c14de890bf1efdb0611dde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.064529 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 2593db823e974f1eb653470d5a23338f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.066394 882 tablet_service.cc:1515] Processing DeleteTablet for tablet bdc5395c1de349b9b20d28625b8751f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.066608 895 tablet_service.cc:1468] Processing CreateTablet for tablet a33c304d652e4e61a3bcfb3ab7067244 (DEFAULT_TABLE table=default.Table_2a80a6c4ae994eda8c3f3fb7b126616c [id=03b41b1e711a4af484ad35073f456efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.067021 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33c304d652e4e61a3bcfb3ab7067244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.070160 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 913bc2f716c14de890bf1efdb0611dde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.070389 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b129117db064a15b27e82439637cf22 with delete_type TABLET_DATA_DELETED (replaced by tablet 35aedb3b590d4917bac8fe0962d68fb3) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.070484 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 063a793e0d7747668d2728fdc2160b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.070587 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet be3313ec5dfb41b68662d3ff4c8c94f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.070386 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 2593db823e974f1eb653470d5a23338f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.070837 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 285912ccbf5a4e5bab53c48d1c024bb8 (DEFAULT_TABLE table=default.Table_0977431f341c4a25825dd7be66a01913 [id=ffb19ea26cba49fa999217f71a4132d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.070930 1014 tablet_service.cc:1468] Processing CreateTablet for tablet a2151ff900b042399199081137c31fd0 (DEFAULT_TABLE table=default.Table_7a3b5892cacf49f29167f2125d14f34a [id=3a329f34821443e6beb7b59680575dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.071244 1017 tablet_service.cc:1468] Processing CreateTablet for tablet d6ce665e69c64e45bac0bdef14f57c3b (DEFAULT_TABLE table=default.Table_3f19c3d36661484dac5b43b45c74851c [id=07bf9483068c420f8be59b01b93e5bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.071422 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c9d010744e88406a828c6ba7b76eda3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.071552 888 tablet_service.cc:1468] Processing CreateTablet for tablet d6ce665e69c64e45bac0bdef14f57c3b (DEFAULT_TABLE table=default.Table_3f19c3d36661484dac5b43b45c74851c [id=07bf9483068c420f8be59b01b93e5bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.071612 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 234be8f208324ddd969555a2e88e2546 (DEFAULT_TABLE table=default.Table_521bb02a0fa64038b747db4f73fa8c90 [id=d1f78b212b5b4608b3bd1edcd80ed747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.071908 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ce665e69c64e45bac0bdef14f57c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.071940 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 80025a9dbde648a29a9a75ca91088634 (DEFAULT_TABLE table=default.Table_7c5253f2510845c58dc974a5a0840b62 [id=37fda201e6e54220873b8631039f6765]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.072121 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 54ea4c4d7756433d85fe9b816b4816ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.072271 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 2191b836bb3044c89fee13c869d9e07d (DEFAULT_TABLE table=default.Table_afac01f597474109bbc2f29c5279b1fa [id=60d58942d03b4b04be7f96c08f758a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.072474 1030 tablet_service.cc:1468] Processing CreateTablet for tablet bbd3200bd1384187978aeb7089ec4e03 (DEFAULT_TABLE table=default.Table_4215c08737a746b59dd7eb42a6a34ba5 [id=ef0b6416799545eba962b9bf74f5256f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.072595 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_847080232892457e98242244a03c75f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.072597 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 82b8afc20aa446e3a714000954f99b0e (DEFAULT_TABLE table=default.Table_d5806194428c4cb9927a09061a2d6533 [id=2c3f0413818e41358b7952e366951cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.072705 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b0037d09796455cbaaadfaa9c010673 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.072811 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.072885 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 067272869c5547648c90f83a34a036aa with delete_type TABLET_DATA_DELETED (replaced by tablet 6340de862fd04c12bb7f85557cfe53d2) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.072957 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dd84b8afa2548fe8ce5925f0bf806f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.073024 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 9302ac10a29c4b0c82268eb767615989 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.072284 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 with delete_type TABLET_DATA_DELETED (replaced by tablet d60228a7b8b7496b89733753ec0b88d3) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.073191 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a33c304d652e4e61a3bcfb3ab7067244 (DEFAULT_TABLE table=default.Table_2a80a6c4ae994eda8c3f3fb7b126616c [id=03b41b1e711a4af484ad35073f456efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.073827 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6024520893584b229315b8f2b59329e0 (DEFAULT_TABLE table=default.Table_943353535aac4d8085a467fcadb1af36 [id=a2891068fff94edb8e9bdd7614002107]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.074820 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f62269079e6d41649f9d7b4c42429ea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.075524 879 tablet_service.cc:1468] Processing CreateTablet for tablet 234be8f208324ddd969555a2e88e2546 (DEFAULT_TABLE table=default.Table_521bb02a0fa64038b747db4f73fa8c90 [id=d1f78b212b5b4608b3bd1edcd80ed747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.075872 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234be8f208324ddd969555a2e88e2546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.078864 1165 ts_tablet_manager.cc:1918] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:12.079061 1165 log.cc:1199] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/cbe4046742b34f1ab0b1610b5d6ee01e
I20250625 13:52:12.079548 883 tablet_service.cc:1468] Processing CreateTablet for tablet 80025a9dbde648a29a9a75ca91088634 (DEFAULT_TABLE table=default.Table_7c5253f2510845c58dc974a5a0840b62 [id=37fda201e6e54220873b8631039f6765]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.079910 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80025a9dbde648a29a9a75ca91088634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.080116 1165 ts_tablet_manager.cc:1939] T cbe4046742b34f1ab0b1610b5d6ee01e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:12.081668 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet cbe4046742b34f1ab0b1610b5d6ee01e (table default.Table_906b9e1a3a8147fb935f7a96968ed294 [id=317ce1c5a5774d6ca8d00ed976bf8615]) successfully deleted
I20250625 13:52:12.082284 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet bdc5395c1de349b9b20d28625b8751f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.082504 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 9302ac10a29c4b0c82268eb767615989 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.082851 881 tablet_service.cc:1468] Processing CreateTablet for tablet 6024520893584b229315b8f2b59329e0 (DEFAULT_TABLE table=default.Table_943353535aac4d8085a467fcadb1af36 [id=a2891068fff94edb8e9bdd7614002107]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.083195 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6024520893584b229315b8f2b59329e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.083248 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cf73b4ab99c466abcfedf1b200e743b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.083527 896 tablet_service.cc:1468] Processing CreateTablet for tablet bbd3200bd1384187978aeb7089ec4e03 (DEFAULT_TABLE table=default.Table_4215c08737a746b59dd7eb42a6a34ba5 [id=ef0b6416799545eba962b9bf74f5256f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.083838 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd3200bd1384187978aeb7089ec4e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.087445 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 with delete_type TABLET_DATA_DELETED (replaced by tablet d60228a7b8b7496b89733753ec0b88d3) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.087541 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aba4565f0af54d4b984435978a9c2821" } new_table_name: "default.Table_b1887d3ec2f14f01808a7cecfa39064f" modify_external_catalogs: true
I20250625 13:52:12.087672 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 54ea4c4d7756433d85fe9b816b4816ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.087879 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.088194 893 tablet_service.cc:1468] Processing CreateTablet for tablet 82b8afc20aa446e3a714000954f99b0e (DEFAULT_TABLE table=default.Table_d5806194428c4cb9927a09061a2d6533 [id=2c3f0413818e41358b7952e366951cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.088552 1166 tablet_bootstrap.cc:492] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:12.088610 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b8afc20aa446e3a714000954f99b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.088711 1166 ts_tablet_manager.cc:1397] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250625 13:52:12.089681 1166 raft_consensus.cc:357] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.089902 1166 raft_consensus.cc:383] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.089984 1166 raft_consensus.cc:738] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.090227 1166 consensus_queue.cc:260] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.090713 1166 ts_tablet_manager.cc:1428] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:12.091008 1166 tablet_bootstrap.cc:492] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:12.092000 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2191b836bb3044c89fee13c869d9e07d (DEFAULT_TABLE table=default.Table_afac01f597474109bbc2f29c5279b1fa [id=60d58942d03b4b04be7f96c08f758a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.092370 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2191b836bb3044c89fee13c869d9e07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.095552 889 tablet_service.cc:1515] Processing DeleteTablet for tablet c9d010744e88406a828c6ba7b76eda3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.095679 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:12.096257 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.096266 884 tablet_service.cc:1468] Processing CreateTablet for tablet 285912ccbf5a4e5bab53c48d1c024bb8 (DEFAULT_TABLE table=default.Table_0977431f341c4a25825dd7be66a01913 [id=ffb19ea26cba49fa999217f71a4132d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.096611 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 285912ccbf5a4e5bab53c48d1c024bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.096897 1165 tablet_replica.cc:331] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:12.099016 1165 raft_consensus.cc:2241] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.099254 1165 raft_consensus.cc:2270] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.099473 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 77c50e1ebb9042ef8f0a651ba7186b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.066409 891 tablet_service.cc:1468] Processing CreateTablet for tablet a2151ff900b042399199081137c31fd0 (DEFAULT_TABLE table=default.Table_7a3b5892cacf49f29167f2125d14f34a [id=3a329f34821443e6beb7b59680575dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.100008 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2151ff900b042399199081137c31fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.100282 1165 ts_tablet_manager.cc:1905] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.103869 1166 tablet_bootstrap.cc:654] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.106369 1165 ts_tablet_manager.cc:1918] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:12.106495 1165 log.cc:1199] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/37e4bd4aaab143ffa65a668a952ae345
I20250625 13:52:12.106860 1165 ts_tablet_manager.cc:1939] T 37e4bd4aaab143ffa65a668a952ae345 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:12.107666 2210 raft_consensus.cc:491] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.107867 2210 raft_consensus.cc:513] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.108480 2210 leader_election.cc:290] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:12.110044 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ba1338d0562418eadabdb8dbcaedee2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:12.110910 851 leader_election.cc:343] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7ba1338d0562418eadabdb8dbcaedee2
W20250625 13:52:12.111505 849 leader_election.cc:343] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.111629 849 leader_election.cc:304] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:12.112028 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_01e8a56cd3c748c2bf70572909dd7215" } modify_external_catalogs: true
I20250625 13:52:12.112272 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_01e8a56cd3c748c2bf70572909dd7215" } modify_external_catalogs: true
I20250625 13:52:12.112716 2204 raft_consensus.cc:2747] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:12.119339 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f4fee43fefb74cd5bd6e7cc25308e969" } modify_external_catalogs: true
I20250625 13:52:12.119534 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f4fee43fefb74cd5bd6e7cc25308e969" } modify_external_catalogs: true
I20250625 13:52:12.119642 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_800985c199634630bea2fe2713086182"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.120185 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_800985c199634630bea2fe2713086182 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.121526 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8be12e4fef05422cb9525f9041369456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.122088 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8be12e4fef05422cb9525f9041369456 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:12.124063 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a079e985358040e78455ac8cb6e49f16 consensus state has no leader
I20250625 13:52:12.124503 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
I20250625 13:52:12.129002 889 tablet_service.cc:1468] Processing CreateTablet for tablet 5395ac846dd649b6885c713026065eec (DEFAULT_TABLE table=default.Table_ecdaa626099d4adaa1f78a0ea582bfa8 [id=453abb550d3d4d6694a3eeae15012900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.129276 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1880ddedc4f54bc5a71c6ef476b57339 (DEFAULT_TABLE table=default.Table_6391ab0c8b3544d0b92e70aa13402edd [id=ba4be2391de44817bbb6fcabee2e58f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.129420 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5395ac846dd649b6885c713026065eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.129666 1012 tablet_service.cc:1468] Processing CreateTablet for tablet a2b98811ad994b8ebce69ea892fd838b (DEFAULT_TABLE table=default.Table_f5e71dc7423c41da9b00ff656cb53b90 [id=d543095910dd409fadcf6d0ae058610f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.129695 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 40f7656d6e53489ab982ba96bc04771b (DEFAULT_TABLE table=default.Table_163659f4838a45f698a8407f2ec827f3 [id=4b6e8cb4d8a3480a999c0bb41a85a58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.130129 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a7b72345e04445f7a393c469a348d126 (DEFAULT_TABLE table=default.Table_4293c9800b3c40498bd0d764bf20c537 [id=7cb87fce00344c7ca642d68bd115afe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.130411 1025 tablet_service.cc:1468] Processing CreateTablet for tablet c9161a0224884c06a496d77b2ba29c78 (DEFAULT_TABLE table=default.Table_3a157223727844e1be7e975b3edd19fe [id=df72a5e675674f359b0d829f46460605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.130438 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 5395ac846dd649b6885c713026065eec (DEFAULT_TABLE table=default.Table_ecdaa626099d4adaa1f78a0ea582bfa8 [id=453abb550d3d4d6694a3eeae15012900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.131492 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 4cec77e7cc9a4108a002e7b7ae45abcb (DEFAULT_TABLE table=default.Table_786e38e4db2a4cc1a486f0f8bc54da13 [id=a1d8270789544b1d9ad869e1f1dd9781]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.133085 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e75bb35262df49c2bd68ad00b19733fd (DEFAULT_TABLE table=default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0 [id=63ad0995448746079f9b85d82eaae0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.134763 881 tablet_service.cc:1468] Processing CreateTablet for tablet 40f7656d6e53489ab982ba96bc04771b (DEFAULT_TABLE table=default.Table_163659f4838a45f698a8407f2ec827f3 [id=4b6e8cb4d8a3480a999c0bb41a85a58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.135255 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f7656d6e53489ab982ba96bc04771b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.136871 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 3c09eb0978274847b54e5665fdfae4f9 (DEFAULT_TABLE table=default.Table_41e5ba0098654109a19569d17b1ab1b9 [id=a5e8e15470a74edab930b32dc641d533]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.138057 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 285912ccbf5a4e5bab53c48d1c024bb8
I20250625 13:52:12.138375 893 tablet_service.cc:1468] Processing CreateTablet for tablet a2b98811ad994b8ebce69ea892fd838b (DEFAULT_TABLE table=default.Table_f5e71dc7423c41da9b00ff656cb53b90 [id=d543095910dd409fadcf6d0ae058610f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.139261 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4111b95923ed4d73b7eca60b964d9494
I20250625 13:52:12.140758 879 tablet_service.cc:1468] Processing CreateTablet for tablet c9161a0224884c06a496d77b2ba29c78 (DEFAULT_TABLE table=default.Table_3a157223727844e1be7e975b3edd19fe [id=df72a5e675674f359b0d829f46460605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.142282 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6110bcce59ea452cbc6e9f464483bde9
I20250625 13:52:12.142699 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ea383a1ceddc461fb2531dbca714029e
I20250625 13:52:12.143625 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet c27caba6c4d14d51b10bf04f131c6a3c
I20250625 13:52:12.141428 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b98811ad994b8ebce69ea892fd838b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.145190 2254 tablet_replica.cc:331] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:12.145387 2254 raft_consensus.cc:2241] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.129402 884 tablet_service.cc:1468] Processing CreateTablet for tablet a7b72345e04445f7a393c469a348d126 (DEFAULT_TABLE table=default.Table_4293c9800b3c40498bd0d764bf20c537 [id=7cb87fce00344c7ca642d68bd115afe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.145574 2254 raft_consensus.cc:2270] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.145936 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 285912ccbf5a4e5bab53c48d1c024bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.146628 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9161a0224884c06a496d77b2ba29c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.147580 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2151ff900b042399199081137c31fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.148042 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7b72345e04445f7a393c469a348d126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.147917 877 tablet_service.cc:1468] Processing CreateTablet for tablet 1880ddedc4f54bc5a71c6ef476b57339 (DEFAULT_TABLE table=default.Table_6391ab0c8b3544d0b92e70aa13402edd [id=ba4be2391de44817bbb6fcabee2e58f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.151039 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 37e4bd4aaab143ffa65a668a952ae345 (table default.Table_3ac6f01366104d75a5bf908bc40c3f4f [id=8a9015750b254a9ea6a48a9510aeddf4]) successfully deleted
I20250625 13:52:12.150769 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1880ddedc4f54bc5a71c6ef476b57339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.151515 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ce665e69c64e45bac0bdef14f57c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.155575 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234be8f208324ddd969555a2e88e2546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.159871 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80025a9dbde648a29a9a75ca91088634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.163498 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2191b836bb3044c89fee13c869d9e07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.167583 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbd3200bd1384187978aeb7089ec4e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.171588 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b8afc20aa446e3a714000954f99b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.175544 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33c304d652e4e61a3bcfb3ab7067244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.179553 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6024520893584b229315b8f2b59329e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.183538 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1880ddedc4f54bc5a71c6ef476b57339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.113903 756 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf7fab53cae49908d2952cdaa07943a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.126369 749 tablet_service.cc:1468] Processing CreateTablet for tablet e75bb35262df49c2bd68ad00b19733fd (DEFAULT_TABLE table=default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0 [id=63ad0995448746079f9b85d82eaae0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.127964 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1880ddedc4f54bc5a71c6ef476b57339 (DEFAULT_TABLE table=default.Table_6391ab0c8b3544d0b92e70aa13402edd [id=ba4be2391de44817bbb6fcabee2e58f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.128270 747 tablet_service.cc:1468] Processing CreateTablet for tablet 40f7656d6e53489ab982ba96bc04771b (DEFAULT_TABLE table=default.Table_163659f4838a45f698a8407f2ec827f3 [id=4b6e8cb4d8a3480a999c0bb41a85a58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.128599 750 tablet_service.cc:1468] Processing CreateTablet for tablet a7b72345e04445f7a393c469a348d126 (DEFAULT_TABLE table=default.Table_4293c9800b3c40498bd0d764bf20c537 [id=7cb87fce00344c7ca642d68bd115afe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.187886 1165 tablet_replica.cc:331] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:12.128747 754 tablet_service.cc:1468] Processing CreateTablet for tablet a2b98811ad994b8ebce69ea892fd838b (DEFAULT_TABLE table=default.Table_f5e71dc7423c41da9b00ff656cb53b90 [id=d543095910dd409fadcf6d0ae058610f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.128911 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5395ac846dd649b6885c713026065eec (DEFAULT_TABLE table=default.Table_ecdaa626099d4adaa1f78a0ea582bfa8 [id=453abb550d3d4d6694a3eeae15012900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.188033 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b98811ad994b8ebce69ea892fd838b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.188083 1165 raft_consensus.cc:2241] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.188241 1165 raft_consensus.cc:2270] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.130828 743 tablet_service.cc:1468] Processing CreateTablet for tablet c9161a0224884c06a496d77b2ba29c78 (DEFAULT_TABLE table=default.Table_3a157223727844e1be7e975b3edd19fe [id=df72a5e675674f359b0d829f46460605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.131569 748 tablet_service.cc:1468] Processing CreateTablet for tablet 4cec77e7cc9a4108a002e7b7ae45abcb (DEFAULT_TABLE table=default.Table_786e38e4db2a4cc1a486f0f8bc54da13 [id=a1d8270789544b1d9ad869e1f1dd9781]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.134239 1166 tablet_bootstrap.cc:492] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:12.188735 1166 ts_tablet_manager.cc:1397] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.098s user 0.005s sys 0.000s
I20250625 13:52:12.189709 1166 raft_consensus.cc:357] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.189914 1166 raft_consensus.cc:383] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.189991 1166 raft_consensus.cc:738] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.190222 1166 consensus_queue.cc:260] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.190819 1166 ts_tablet_manager.cc:1428] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:12.135418 741 tablet_service.cc:1468] Processing CreateTablet for tablet 3c09eb0978274847b54e5665fdfae4f9 (DEFAULT_TABLE table=default.Table_41e5ba0098654109a19569d17b1ab1b9 [id=a5e8e15470a74edab930b32dc641d533]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.138758 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 285912ccbf5a4e5bab53c48d1c024bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.139966 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 4111b95923ed4d73b7eca60b964d9494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.140826 755 tablet_service.cc:1468] Processing CreateTablet for tablet d60228a7b8b7496b89733753ec0b88d3 (DEFAULT_TABLE table=default.Table_477135ae07a343758db03b263574b92b [id=a9e36e91e1ae4695a68daad3d1cada49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.143144 745 tablet_service.cc:1468] Processing CreateTablet for tablet 14087755cd5744958499f32fd77b655b (DEFAULT_TABLE table=default.Table_ed99b177ce51470a970fe7a1d9e6e25e [id=b97b78be08fe4923b40935d139846423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.143723 758 tablet_service.cc:1515] Processing DeleteTablet for tablet ea383a1ceddc461fb2531dbca714029e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.144203 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 6110bcce59ea452cbc6e9f464483bde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.144572 746 tablet_service.cc:1515] Processing DeleteTablet for tablet c27caba6c4d14d51b10bf04f131c6a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.191375 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1880ddedc4f54bc5a71c6ef476b57339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.191497 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f7656d6e53489ab982ba96bc04771b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.194892 1165 ts_tablet_manager.cc:1905] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.195863 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7b72345e04445f7a393c469a348d126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.196470 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60228a7b8b7496b89733753ec0b88d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.199528 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9161a0224884c06a496d77b2ba29c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.199542 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b98811ad994b8ebce69ea892fd838b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.203516 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5395ac846dd649b6885c713026065eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.203558 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5395ac846dd649b6885c713026065eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.207567 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cec77e7cc9a4108a002e7b7ae45abcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.207696 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f7656d6e53489ab982ba96bc04771b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.207930 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14087755cd5744958499f32fd77b655b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.207955 1166 tablet_bootstrap.cc:492] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:12.210161 1166 tablet_bootstrap.cc:654] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.211522 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75bb35262df49c2bd68ad00b19733fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.212653 1165 ts_tablet_manager.cc:1918] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:12.212765 1165 log.cc:1199] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/dcf7fab53cae49908d2952cdaa07943a
I20250625 13:52:12.213063 1165 ts_tablet_manager.cc:1939] T dcf7fab53cae49908d2952cdaa07943a P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:12.213706 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7b72345e04445f7a393c469a348d126. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.215569 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c09eb0978274847b54e5665fdfae4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.220033 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1e94167cff8f4ee8ad52fa7f9e012a35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.220544 2254 ts_tablet_manager.cc:1905] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:12.220667 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e94167cff8f4ee8ad52fa7f9e012a35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.221448 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet dcf7fab53cae49908d2952cdaa07943a (table default.Table_fe872dae10544c0ca6a6735266cd03d2 [id=aa4ef3fe7bbb408396878d79f7caffb9]) successfully deleted
I20250625 13:52:12.223788 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cec77e7cc9a4108a002e7b7ae45abcb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.224256 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 913bc2f716c14de890bf1efdb0611dde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913bc2f716c14de890bf1efdb0611dde already in progress: creating tablet
I20250625 13:52:12.224547 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 285912ccbf5a4e5bab53c48d1c024bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.224892 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 14087755cd5744958499f32fd77b655b (DEFAULT_TABLE table=default.Table_ed99b177ce51470a970fe7a1d9e6e25e [id=b97b78be08fe4923b40935d139846423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.225255 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14087755cd5744958499f32fd77b655b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.227483 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9161a0224884c06a496d77b2ba29c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.231696 2210 raft_consensus.cc:491] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.231849 2210 raft_consensus.cc:513] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.232411 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_163659f4838a45f698a8407f2ec827f3" } new_table_name: "default.Table_f2fba4b44d8e4e01994d0e99459c25f6" modify_external_catalogs: true
I20250625 13:52:12.232474 2210 leader_election.cc:290] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:12.232484 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2cc08863cc3d48d99668f2fac71fa727"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.233034 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cc08863cc3d48d99668f2fac71fa727 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.233243 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c50e1ebb9042ef8f0a651ba7186b74" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:12.233417 1047 raft_consensus.cc:2466] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:12.233629 851 leader_election.cc:343] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 77c50e1ebb9042ef8f0a651ba7186b74
I20250625 13:52:12.233920 849 leader_election.cc:304] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:12.234212 2210 raft_consensus.cc:2802] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:12.234150 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_da70d46ae8854be99605668d2d0fdf95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.234301 2210 raft_consensus.cc:491] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:12.234385 2210 raft_consensus.cc:3058] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:12.232173 1166 tablet_bootstrap.cc:492] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
W20250625 13:52:12.234587 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da70d46ae8854be99605668d2d0fdf95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.234766 1166 ts_tablet_manager.cc:1397] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20250625 13:52:12.235888 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e3a43a004fde420b9c388ab931d7e3eb" } new_table_name: "default.Table_3f5dc7a3e577405391f6d3ec22c932ff" modify_external_catalogs: true
I20250625 13:52:12.236202 1166 raft_consensus.cc:357] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.236378 1166 raft_consensus.cc:383] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.236450 1166 raft_consensus.cc:738] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.236673 1166 consensus_queue.cc:260] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.237190 1166 ts_tablet_manager.cc:1428] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:12.237221 2210 raft_consensus.cc:513] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.239189 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_679758c0160345d48f8a3bfc5a1a042d" } new_table_name: "default.Table_fd2fbb332353449080133a5788a93d74" modify_external_catalogs: true
I20250625 13:52:12.240711 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77c50e1ebb9042ef8f0a651ba7186b74" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:12.240876 1047 raft_consensus.cc:3058] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:12.241861 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 4111b95923ed4d73b7eca60b964d9494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.242231 1011 tablet_service.cc:1468] Processing CreateTablet for tablet d60228a7b8b7496b89733753ec0b88d3 (DEFAULT_TABLE table=default.Table_477135ae07a343758db03b263574b92b [id=a9e36e91e1ae4695a68daad3d1cada49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.242576 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60228a7b8b7496b89733753ec0b88d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.243389 1047 raft_consensus.cc:2466] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
W20250625 13:52:12.243748 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 64793.7us
I20250625 13:52:12.244138 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 7efc8df3c8e24020b073a1b61bc9e0a0 from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:12.245834 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 40f7656d6e53489ab982ba96bc04771b consensus state has no leader
I20250625 13:52:12.248731 849 leader_election.cc:304] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:12.249163 2204 raft_consensus.cc:2802] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:12.249317 2204 raft_consensus.cc:695] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:12.249642 2204 consensus_queue.cc:237] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:12.252089 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ac647b1d704142b94da8d506319466 consensus state has no leader
I20250625 13:52:12.252370 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 6110bcce59ea452cbc6e9f464483bde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.252575 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet ea383a1ceddc461fb2531dbca714029e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:12.253306 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8257f79a42247b48f2736a545362f44 consensus state has no leader
I20250625 13:52:12.255041 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f13b94122bf947aab60dbf0eb035cc63" } new_table_name: "default.Table_e17cf60c94784737b39faf63421fff96" modify_external_catalogs: true
W20250625 13:52:12.259799 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 285912ccbf5a4e5bab53c48d1c024bb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 285912ccbf5a4e5bab53c48d1c024bb8 already in progress: creating tablet
W20250625 13:52:12.261016 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 4111b95923ed4d73b7eca60b964d9494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4111b95923ed4d73b7eca60b964d9494 already in progress: creating tablet
I20250625 13:52:12.261535 1166 tablet_bootstrap.cc:492] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:12.266558 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75bb35262df49c2bd68ad00b19733fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.267568 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ea383a1ceddc461fb2531dbca714029e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea383a1ceddc461fb2531dbca714029e already in progress: creating tablet
W20250625 13:52:12.267784 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6110bcce59ea452cbc6e9f464483bde9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6110bcce59ea452cbc6e9f464483bde9 already in progress: creating tablet
W20250625 13:52:12.267944 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet c27caba6c4d14d51b10bf04f131c6a3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c27caba6c4d14d51b10bf04f131c6a3c already in progress: creating tablet
I20250625 13:52:12.267323 1166 tablet_bootstrap.cc:654] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.268576 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3859f399a7b84de99658380bd6c24ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.269637 1165 tablet_replica.cc:331] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:12.269833 1165 raft_consensus.cc:2241] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.269992 1165 raft_consensus.cc:2270] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.270695 1165 ts_tablet_manager.cc:1905] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.274080 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c09eb0978274847b54e5665fdfae4f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.279065 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23a2b56c117f49d3b7bab4bb3d776173 consensus state has no leader
I20250625 13:52:12.280408 1165 ts_tablet_manager.cc:1918] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:12.280565 1165 log.cc:1199] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/3859f399a7b84de99658380bd6c24ff5
I20250625 13:52:12.280900 1165 ts_tablet_manager.cc:1939] T 3859f399a7b84de99658380bd6c24ff5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:12.281800 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 3859f399a7b84de99658380bd6c24ff5 (table default.Table_cf29f4b4bbb94091affd4440a4b8447c [id=27acc786d57a4c7ab531242dd52f17ab]) successfully deleted
I20250625 13:52:12.296018 884 tablet_service.cc:1468] Processing CreateTablet for tablet d60228a7b8b7496b89733753ec0b88d3 (DEFAULT_TABLE table=default.Table_477135ae07a343758db03b263574b92b [id=a9e36e91e1ae4695a68daad3d1cada49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.296603 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60228a7b8b7496b89733753ec0b88d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.296984 889 tablet_service.cc:1515] Processing DeleteTablet for tablet ea383a1ceddc461fb2531dbca714029e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.255261 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet c27caba6c4d14d51b10bf04f131c6a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.297468 889 tablet_service.cc:1468] Processing CreateTablet for tablet 14087755cd5744958499f32fd77b655b (DEFAULT_TABLE table=default.Table_ed99b177ce51470a970fe7a1d9e6e25e [id=b97b78be08fe4923b40935d139846423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.297890 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14087755cd5744958499f32fd77b655b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.298408 895 tablet_service.cc:1515] Processing DeleteTablet for tablet c27caba6c4d14d51b10bf04f131c6a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.298695 2210 leader_election.cc:290] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:12.299050 851 leader_election.cc:343] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 77c50e1ebb9042ef8f0a651ba7186b74
I20250625 13:52:12.144084 891 tablet_service.cc:1468] Processing CreateTablet for tablet 4cec77e7cc9a4108a002e7b7ae45abcb (DEFAULT_TABLE table=default.Table_786e38e4db2a4cc1a486f0f8bc54da13 [id=a1d8270789544b1d9ad869e1f1dd9781]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.299417 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cec77e7cc9a4108a002e7b7ae45abcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.144367 896 tablet_service.cc:1468] Processing CreateTablet for tablet 3c09eb0978274847b54e5665fdfae4f9 (DEFAULT_TABLE table=default.Table_41e5ba0098654109a19569d17b1ab1b9 [id=a5e8e15470a74edab930b32dc641d533]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.143630 888 tablet_service.cc:1468] Processing CreateTablet for tablet e75bb35262df49c2bd68ad00b19733fd (DEFAULT_TABLE table=default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0 [id=63ad0995448746079f9b85d82eaae0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.295523 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 285912ccbf5a4e5bab53c48d1c024bb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:11 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.304095 1166 tablet_bootstrap.cc:492] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:12.304248 1166 ts_tablet_manager.cc:1397] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20250625 13:52:12.305060 1166 raft_consensus.cc:357] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.305243 1166 raft_consensus.cc:383] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.305325 1166 raft_consensus.cc:738] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.305577 1166 consensus_queue.cc:260] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.291205 2254 ts_tablet_manager.cc:1918] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:12.306185 1166 ts_tablet_manager.cc:1428] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 13:52:12.306185 2254 log.cc:1199] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/0b129117db064a15b27e82439637cf22
I20250625 13:52:12.306535 1166 tablet_bootstrap.cc:492] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:12.306603 2254 ts_tablet_manager.cc:1939] T 0b129117db064a15b27e82439637cf22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:12.307822 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 0b129117db064a15b27e82439637cf22 (table default.Table_78be58e64e564e6ab48ccee86fbe5279 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]) successfully deleted
I20250625 13:52:12.308166 1166 tablet_bootstrap.cc:654] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.309376 2254 tablet_replica.cc:331] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:12.309796 2254 raft_consensus.cc:2241] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:12.310385 2254 raft_consensus.cc:2270] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.312382 2254 ts_tablet_manager.cc:1905] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.316165 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.316974 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bbc520f4e79e430b9629ee6fe8682d1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.317620 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc520f4e79e430b9629ee6fe8682d1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.318035 1165 tablet_replica.cc:331] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:12.318238 1165 raft_consensus.cc:2241] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.318395 1165 raft_consensus.cc:2270] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.319008 1165 ts_tablet_manager.cc:1905] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.324911 1165 ts_tablet_manager.cc:1918] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:12.325480 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_957dc0554e3040d38f97ccfa311f4270"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.325985 1165 log.cc:1199] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/9514bbbda0ba4329a18b0a42e635c6e5
W20250625 13:52:12.326236 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_957dc0554e3040d38f97ccfa311f4270 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.326378 1165 ts_tablet_manager.cc:1939] T 9514bbbda0ba4329a18b0a42e635c6e5 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:12.327383 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 9514bbbda0ba4329a18b0a42e635c6e5 (table default.Table_3462a32d0970423c8538a79dae984471 [id=60ccb9d481c14fd68670e5701ce76b84]) successfully deleted
I20250625 13:52:12.328637 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 6110bcce59ea452cbc6e9f464483bde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.328828 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 4111b95923ed4d73b7eca60b964d9494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.329308 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c09eb0978274847b54e5665fdfae4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.329445 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6ad4a7572a6941ca8392da904cd328bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.329965 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ad4a7572a6941ca8392da904cd328bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.330626 1171 tablet_bootstrap.cc:492] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:12.330760 1171 ts_tablet_manager.cc:1397] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.315s user 0.001s sys 0.001s
I20250625 13:52:12.331888 1171 raft_consensus.cc:357] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.332089 1171 raft_consensus.cc:383] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.332166 1171 raft_consensus.cc:738] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.335559 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6755f077e9a0460d86f174be4593542a" } new_table_name: "default.Table_7cffeb8b84ab45ae9278474f3ed5107f" modify_external_catalogs: true
W20250625 13:52:12.343202 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e38a85c299f04d60a3c465571fd33971 consensus state has no leader
I20250625 13:52:12.347432 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d2d334c69d834f9d89194a4ce4dc2fdd" } modify_external_catalogs: true
I20250625 13:52:12.347640 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d2d334c69d834f9d89194a4ce4dc2fdd" } modify_external_catalogs: true
I20250625 13:52:12.348115 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_583f49dbf61c498b98a3cd236437318d" } modify_external_catalogs: true
I20250625 13:52:12.348233 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e75bb35262df49c2bd68ad00b19733fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.348268 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_583f49dbf61c498b98a3cd236437318d" } modify_external_catalogs: true
I20250625 13:52:12.348732 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ff276efb7beb42dcaf8cb45ca0c0e04a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.349253 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff276efb7beb42dcaf8cb45ca0c0e04a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.350345 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0ca3566f743046db9a91f2a4040f2e03" } modify_external_catalogs: true
I20250625 13:52:12.350477 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0ca3566f743046db9a91f2a4040f2e03" } modify_external_catalogs: true
I20250625 13:52:12.350940 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a3634181d607460c9e84b9cca878d57b" } modify_external_catalogs: true
I20250625 13:52:12.351127 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a3634181d607460c9e84b9cca878d57b" } modify_external_catalogs: true
I20250625 13:52:12.351753 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6bb83d62643e419cbe5a1ac4c61fea12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.352258 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bb83d62643e419cbe5a1ac4c61fea12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.354571 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7096ca87d4754bec97389b249556e073"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.355037 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7096ca87d4754bec97389b249556e073 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.357510 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.358047 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.359030 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 05364e23e0ec4cffb92a8df2d38ca4b9 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:12.361052 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 05364e23e0ec4cffb92a8df2d38ca4b9 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:12.365446 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a5dde0fa6d7c471f983b3c8e01eabbcd" } modify_external_catalogs: true
I20250625 13:52:12.365667 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a5dde0fa6d7c471f983b3c8e01eabbcd" } modify_external_catalogs: true
I20250625 13:52:12.369457 1171 consensus_queue.cc:260] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.370601 1018 tablet_service.cc:1468] Processing CreateTablet for tablet dab017faa5d44a71a1ac2ee0e5776bfb (DEFAULT_TABLE table=default.Table_6fa536f6af5e444abbb620755a21d5c0 [id=bf5ec95f5519458aae5df2b8cf7a78bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.370711 749 tablet_service.cc:1468] Processing CreateTablet for tablet dab017faa5d44a71a1ac2ee0e5776bfb (DEFAULT_TABLE table=default.Table_6fa536f6af5e444abbb620755a21d5c0 [id=bf5ec95f5519458aae5df2b8cf7a78bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.371064 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab017faa5d44a71a1ac2ee0e5776bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.371145 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab017faa5d44a71a1ac2ee0e5776bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.372352 747 tablet_service.cc:1468] Processing CreateTablet for tablet a68b159f1b3d45b6a8931a9ca42939c6 (DEFAULT_TABLE table=default.Table_e64d8f11a6ed4105a26c31a57a80f3aa [id=831ddd1ffacd4947a8ff9f644dde02c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.372699 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68b159f1b3d45b6a8931a9ca42939c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.375588 741 tablet_service.cc:1468] Processing CreateTablet for tablet dabe0ec0b2bc4c92b9744c9b5378bcfb (DEFAULT_TABLE table=default.Table_0566054eac7f44f0801ea96f33d099fd [id=dd6bc29cd1d84f189d4fa6811fab4d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.375964 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabe0ec0b2bc4c92b9744c9b5378bcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.375897 1016 tablet_service.cc:1468] Processing CreateTablet for tablet e597f8cd8e5a4ec6be8db57f65c53c65 (DEFAULT_TABLE table=default.Table_3e4c58462d2844a6be928491924b59fb [id=f1a226fffbaf46a39cbf58aad8f429ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.376245 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e597f8cd8e5a4ec6be8db57f65c53c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.379588 742 tablet_service.cc:1468] Processing CreateTablet for tablet e597f8cd8e5a4ec6be8db57f65c53c65 (DEFAULT_TABLE table=default.Table_3e4c58462d2844a6be928491924b59fb [id=f1a226fffbaf46a39cbf58aad8f429ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.379719 1012 tablet_service.cc:1468] Processing CreateTablet for tablet a68b159f1b3d45b6a8931a9ca42939c6 (DEFAULT_TABLE table=default.Table_e64d8f11a6ed4105a26c31a57a80f3aa [id=831ddd1ffacd4947a8ff9f644dde02c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.379923 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e597f8cd8e5a4ec6be8db57f65c53c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.380059 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68b159f1b3d45b6a8931a9ca42939c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.380142 2279 raft_consensus.cc:491] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.380265 2279 raft_consensus.cc:513] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.380903 2279 leader_election.cc:290] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:12.381834 985 leader_election.cc:343] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 5cf73b4ab99c466abcfedf1b200e743b
I20250625 13:52:12.384229 1021 tablet_service.cc:1468] Processing CreateTablet for tablet dabe0ec0b2bc4c92b9744c9b5378bcfb (DEFAULT_TABLE table=default.Table_0566054eac7f44f0801ea96f33d099fd [id=dd6bc29cd1d84f189d4fa6811fab4d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.384603 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabe0ec0b2bc4c92b9744c9b5378bcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.386839 743 tablet_service.cc:1468] Processing CreateTablet for tablet 0643aecdc83a4fcaa3b1e31f029d3627 (DEFAULT_TABLE table=default.Table_d3ebd3c017504031acb0eb2ffa156cdf [id=80af6a4fcd924b72b207c1e53b5649a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.387228 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0643aecdc83a4fcaa3b1e31f029d3627. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.387427 983 leader_election.cc:343] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 5cf73b4ab99c466abcfedf1b200e743b
I20250625 13:52:12.387529 983 leader_election.cc:304] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:12.388365 2279 raft_consensus.cc:2747] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:12.388495 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 0643aecdc83a4fcaa3b1e31f029d3627 (DEFAULT_TABLE table=default.Table_d3ebd3c017504031acb0eb2ffa156cdf [id=80af6a4fcd924b72b207c1e53b5649a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.388813 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0643aecdc83a4fcaa3b1e31f029d3627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.391649 1028 tablet_service.cc:1468] Processing CreateTablet for tablet ecd109dacfcb48a7a5ab54ef6f85a1b1 (DEFAULT_TABLE table=default.Table_8be17994b5324638bcceef7134adbaf1 [id=8755c5c4e368440e9e7faf4e05665b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.391821 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2467f6e2b0be4abbaa3555f33e42fa76 (DEFAULT_TABLE table=default.Table_efee3de551ce4c8c96c15ea79ad7fc77 [id=28ff914e41654516a1db0b21d5e28124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.391996 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd109dacfcb48a7a5ab54ef6f85a1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.392170 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2467f6e2b0be4abbaa3555f33e42fa76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.392498 755 tablet_service.cc:1468] Processing CreateTablet for tablet ae16c5d284b6464da1bbb9ca516de821 (DEFAULT_TABLE table=default.Table_b7652d48c11040f19386af4a01ea6be6 [id=b794e918cb9f4ff4b3f29360a5212c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.392871 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae16c5d284b6464da1bbb9ca516de821. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.396056 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 2467f6e2b0be4abbaa3555f33e42fa76 (DEFAULT_TABLE table=default.Table_efee3de551ce4c8c96c15ea79ad7fc77 [id=28ff914e41654516a1db0b21d5e28124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.396425 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2467f6e2b0be4abbaa3555f33e42fa76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.399935 879 tablet_service.cc:1468] Processing CreateTablet for tablet dab017faa5d44a71a1ac2ee0e5776bfb (DEFAULT_TABLE table=default.Table_6fa536f6af5e444abbb620755a21d5c0 [id=bf5ec95f5519458aae5df2b8cf7a78bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.400353 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab017faa5d44a71a1ac2ee0e5776bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.401338 755 tablet_service.cc:1468] Processing CreateTablet for tablet 6b818fd2c00b41a2aaded9f7dbc1b276 (DEFAULT_TABLE table=default.Table_8be12e4fef05422cb9525f9041369456 [id=7b40cb83190146269d8953e96d4e2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.401693 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b818fd2c00b41a2aaded9f7dbc1b276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.404036 894 tablet_service.cc:1468] Processing CreateTablet for tablet ecd109dacfcb48a7a5ab54ef6f85a1b1 (DEFAULT_TABLE table=default.Table_8be17994b5324638bcceef7134adbaf1 [id=8755c5c4e368440e9e7faf4e05665b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.403934 877 tablet_service.cc:1468] Processing CreateTablet for tablet e597f8cd8e5a4ec6be8db57f65c53c65 (DEFAULT_TABLE table=default.Table_3e4c58462d2844a6be928491924b59fb [id=f1a226fffbaf46a39cbf58aad8f429ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.404397 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd109dacfcb48a7a5ab54ef6f85a1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.404686 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e597f8cd8e5a4ec6be8db57f65c53c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.408694 747 tablet_service.cc:1468] Processing CreateTablet for tablet ecd109dacfcb48a7a5ab54ef6f85a1b1 (DEFAULT_TABLE table=default.Table_8be17994b5324638bcceef7134adbaf1 [id=8755c5c4e368440e9e7faf4e05665b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.409168 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd109dacfcb48a7a5ab54ef6f85a1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.409292 749 tablet_service.cc:1468] Processing CreateTablet for tablet 1fc122d92c644e0d908c327c002708fb (DEFAULT_TABLE table=default.Table_847080232892457e98242244a03c75f6 [id=074a3f1be226435a90a11749bb29d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.409631 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc122d92c644e0d908c327c002708fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.411108 885 tablet_service.cc:1468] Processing CreateTablet for tablet 0643aecdc83a4fcaa3b1e31f029d3627 (DEFAULT_TABLE table=default.Table_d3ebd3c017504031acb0eb2ffa156cdf [id=80af6a4fcd924b72b207c1e53b5649a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.411577 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0643aecdc83a4fcaa3b1e31f029d3627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.414499 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet b70e7607709e4be395f1ed8d01d7d169
I20250625 13:52:12.415372 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 05364e23e0ec4cffb92a8df2d38ca4b9
I20250625 13:52:12.415750 591 catalog_manager.cc:6652] ReplaceTablet: tablet 05364e23e0ec4cffb92a8df2d38ca4b9 deleted and replaced by tablet 26530e4206b34be29c7e822aa4ed1c93
I20250625 13:52:12.417028 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 30b5b3444fd245bbb2ecd5889d956862
I20250625 13:52:12.417397 889 tablet_service.cc:1468] Processing CreateTablet for tablet dabe0ec0b2bc4c92b9744c9b5378bcfb (DEFAULT_TABLE table=default.Table_0566054eac7f44f0801ea96f33d099fd [id=dd6bc29cd1d84f189d4fa6811fab4d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.417873 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabe0ec0b2bc4c92b9744c9b5378bcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.418453 742 tablet_service.cc:1515] Processing DeleteTablet for tablet b70e7607709e4be395f1ed8d01d7d169 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.419204 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 26530e4206b34be29c7e822aa4ed1c93) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:12.419364 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet b70e7607709e4be395f1ed8d01d7d169 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b70e7607709e4be395f1ed8d01d7d169 already in progress: creating tablet
W20250625 13:52:12.420063 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05364e23e0ec4cffb92a8df2d38ca4b9 already in progress: creating tablet
I20250625 13:52:12.418685 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 30b5b3444fd245bbb2ecd5889d956862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:12.420770 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 30b5b3444fd245bbb2ecd5889d956862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30b5b3444fd245bbb2ecd5889d956862 already in progress: creating tablet
I20250625 13:52:12.422063 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e438e46874f841d2ac972b953ddb4608
I20250625 13:52:12.422127 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a8c4f90b40c344168e8f689c43101a64
I20250625 13:52:12.423008 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a8c4f90b40c344168e8f689c43101a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.423499 895 tablet_service.cc:1468] Processing CreateTablet for tablet a68b159f1b3d45b6a8931a9ca42939c6 (DEFAULT_TABLE table=default.Table_e64d8f11a6ed4105a26c31a57a80f3aa [id=831ddd1ffacd4947a8ff9f644dde02c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.423604 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e438e46874f841d2ac972b953ddb4608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:12.423870 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a8c4f90b40c344168e8f689c43101a64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c4f90b40c344168e8f689c43101a64 already in progress: creating tablet
I20250625 13:52:12.423938 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68b159f1b3d45b6a8931a9ca42939c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.424136 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e438e46874f841d2ac972b953ddb4608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e438e46874f841d2ac972b953ddb4608 already in progress: creating tablet
I20250625 13:52:12.424557 742 tablet_service.cc:1468] Processing CreateTablet for tablet c6f3c87060a1463d9b6c5b76e531eaf0 (DEFAULT_TABLE table=default.Table_f62269079e6d41649f9d7b4c42429ea3 [id=791501421c3944b6b7f910fcb314fdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.424938 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f3c87060a1463d9b6c5b76e531eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.429919 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a8c4f90b40c344168e8f689c43101a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.430445 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 1fc122d92c644e0d908c327c002708fb (DEFAULT_TABLE table=default.Table_847080232892457e98242244a03c75f6 [id=074a3f1be226435a90a11749bb29d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.430945 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc122d92c644e0d908c327c002708fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.434826 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 30b5b3444fd245bbb2ecd5889d956862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.436040 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 26530e4206b34be29c7e822aa4ed1c93) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.436264 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet b70e7607709e4be395f1ed8d01d7d169 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.435443 1013 tablet_service.cc:1468] Processing CreateTablet for tablet c6f3c87060a1463d9b6c5b76e531eaf0 (DEFAULT_TABLE table=default.Table_f62269079e6d41649f9d7b4c42429ea3 [id=791501421c3944b6b7f910fcb314fdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.436906 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f3c87060a1463d9b6c5b76e531eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.437758 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 34371795c8094e73ab4e5dbeedb7aa2f
I20250625 13:52:12.439018 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 34371795c8094e73ab4e5dbeedb7aa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.435691 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ae16c5d284b6464da1bbb9ca516de821 (DEFAULT_TABLE table=default.Table_b7652d48c11040f19386af4a01ea6be6 [id=b794e918cb9f4ff4b3f29360a5212c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.439422 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 34371795c8094e73ab4e5dbeedb7aa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.439795 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae16c5d284b6464da1bbb9ca516de821. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.440137 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 34371795c8094e73ab4e5dbeedb7aa2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34371795c8094e73ab4e5dbeedb7aa2f already in progress: creating tablet
I20250625 13:52:12.435760 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet e438e46874f841d2ac972b953ddb4608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.443384 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_17343b483b8e4268a6a3cca32bbcb186"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.443837 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6b818fd2c00b41a2aaded9f7dbc1b276 (DEFAULT_TABLE table=default.Table_8be12e4fef05422cb9525f9041369456 [id=7b40cb83190146269d8953e96d4e2483]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.444196 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17343b483b8e4268a6a3cca32bbcb186 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.444291 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b818fd2c00b41a2aaded9f7dbc1b276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.449587 890 tablet_service.cc:1468] Processing CreateTablet for tablet 2467f6e2b0be4abbaa3555f33e42fa76 (DEFAULT_TABLE table=default.Table_efee3de551ce4c8c96c15ea79ad7fc77 [id=28ff914e41654516a1db0b21d5e28124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.450060 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2467f6e2b0be4abbaa3555f33e42fa76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.451910 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e50eae9bd67942f5a09da3884500dac7" } new_table_name: "default.Table_46c504eaed8042dd930600225e3fe0b5" modify_external_catalogs: true
I20250625 13:52:12.454440 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_353eb61ad57d4e8084b18257adf1f9a1" } new_table_name: "default.Table_6e1cdf5a629142a98879f22c244338eb" modify_external_catalogs: true
I20250625 13:52:12.451704 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f5cca2b6860a4245a3c3d183b0143909"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.455657 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fe321782d15943acaf71e83adc22ab28" } new_table_name: "default.Table_d3e067c08fee43e69098a905cf6418aa" modify_external_catalogs: true
I20250625 13:52:12.456056 883 tablet_service.cc:1468] Processing CreateTablet for tablet ae16c5d284b6464da1bbb9ca516de821 (DEFAULT_TABLE table=default.Table_b7652d48c11040f19386af4a01ea6be6 [id=b794e918cb9f4ff4b3f29360a5212c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.456410 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae16c5d284b6464da1bbb9ca516de821. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.455716 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5cca2b6860a4245a3c3d183b0143909 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.459870 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fec914ebb830498da551091c86a181cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.460377 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fec914ebb830498da551091c86a181cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:12.460953 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ec5fe0ee4354f0cbadf2f1be23d8bd9 consensus state has no leader
I20250625 13:52:12.462224 880 tablet_service.cc:1468] Processing CreateTablet for tablet 6b818fd2c00b41a2aaded9f7dbc1b276 (DEFAULT_TABLE table=default.Table_8be12e4fef05422cb9525f9041369456 [id=7b40cb83190146269d8953e96d4e2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.462602 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b818fd2c00b41a2aaded9f7dbc1b276. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.464054 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 consensus state has no leader
W20250625 13:52:12.465149 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd036ec08f774f5393067ce20bb07610 consensus state has no leader
I20250625 13:52:12.468326 882 tablet_service.cc:1515] Processing DeleteTablet for tablet b70e7607709e4be395f1ed8d01d7d169 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.468848 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 26530e4206b34be29c7e822aa4ed1c93) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.469200 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 30b5b3444fd245bbb2ecd5889d956862 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.470635 882 tablet_service.cc:1468] Processing CreateTablet for tablet 1fc122d92c644e0d908c327c002708fb (DEFAULT_TABLE table=default.Table_847080232892457e98242244a03c75f6 [id=074a3f1be226435a90a11749bb29d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.470990 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc122d92c644e0d908c327c002708fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.470927 878 tablet_service.cc:1468] Processing CreateTablet for tablet c6f3c87060a1463d9b6c5b76e531eaf0 (DEFAULT_TABLE table=default.Table_f62269079e6d41649f9d7b4c42429ea3 [id=791501421c3944b6b7f910fcb314fdaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.471199 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f3c87060a1463d9b6c5b76e531eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.471462 887 tablet_service.cc:1515] Processing DeleteTablet for tablet a8c4f90b40c344168e8f689c43101a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.473199 774 raft_consensus.cc:1273] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer affa1c3035b940e88ca143be7d35840d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:12.473769 2210 consensus_queue.cc:1035] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:12.480331 893 tablet_service.cc:1515] Processing DeleteTablet for tablet e438e46874f841d2ac972b953ddb4608 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.480649 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_31c75178c8ab41378e5a92bb7b2ccb9f" } new_table_name: "default.Table_30ad949fc3de40d18eb5ace48fd633c8" modify_external_catalogs: true
I20250625 13:52:12.483762 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 34371795c8094e73ab4e5dbeedb7aa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:12.491520 849 consensus_peers.cc:487] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:12.493129 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0642b80eda44397878897217a29af65 consensus state has no leader
W20250625 13:52:12.498693 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 110513us
I20250625 13:52:12.499929 2303 tablet.cc:1722] T 9267df231f354c578c38f452418e523c P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:12.500648 2266 tablet.cc:1722] T 9267df231f354c578c38f452418e523c P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:12.510917 596 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
I20250625 13:52:12.518381 597 master_service.cc:948] ReplaceTablet: received request to replace tablet e75bb35262df49c2bd68ad00b19733fd from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:12.524181 590 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_635bd0549cf242ff8169101c0efa90b6 [id=dc8adfaa562a41bca5cede19a64dced4] alter complete (version 1)
I20250625 13:52:12.527227 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 490c8902efeb46c99543747a8787c1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.525854 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e2e20a31ea3544cf974a8487d18364e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.528020 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2e20a31ea3544cf974a8487d18364e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.041672 1168 ts_tablet_manager.cc:1905] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.532584 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e75bb35262df49c2bd68ad00b19733fd
I20250625 13:52:12.533124 597 catalog_manager.cc:6652] ReplaceTablet: tablet e75bb35262df49c2bd68ad00b19733fd deleted and replaced by tablet 5e7bf419440645279046b030e7554893
I20250625 13:52:12.533560 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet e75bb35262df49c2bd68ad00b19733fd with delete_type TABLET_DATA_DELETED (replaced by tablet 5e7bf419440645279046b030e7554893) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.534214 893 tablet_service.cc:1515] Processing DeleteTablet for tablet e75bb35262df49c2bd68ad00b19733fd with delete_type TABLET_DATA_DELETED (replaced by tablet 5e7bf419440645279046b030e7554893) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.535465 747 tablet_service.cc:1515] Processing DeleteTablet for tablet e75bb35262df49c2bd68ad00b19733fd with delete_type TABLET_DATA_DELETED (replaced by tablet 5e7bf419440645279046b030e7554893) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:12.536378 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e75bb35262df49c2bd68ad00b19733fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e75bb35262df49c2bd68ad00b19733fd already in progress: creating tablet
I20250625 13:52:12.537704 1168 ts_tablet_manager.cc:1918] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:12.546582 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_945803b681074611b2bfcd213a5e072d" } new_table_name: "default.Table_de9b2bd8421841fab7fff76f350bf461" modify_external_catalogs: true
I20250625 13:52:12.548542 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ef5aa5771b2b412ca7ebce55f40b6eed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.549098 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef5aa5771b2b412ca7ebce55f40b6eed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:12.554767 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62f748645e894531b3b1eb694d45e6b0 consensus state has no leader
I20250625 13:52:12.559461 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_78d7cfce8bd9428c8b6f47e48fc1ddb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.560101 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78d7cfce8bd9428c8b6f47e48fc1ddb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.562075 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b033e3575af14620a4ffd13d75b56bce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.562557 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b033e3575af14620a4ffd13d75b56bce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:12.567040 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 027b1281543b4d559461e8d222080c30 (table default.Table_01c5150924d84a429e9004bb668e4550 [id=dec0d3288d724ffeb90336093522ce51], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 027b1281543b4d559461e8d222080c30 consensus state has no leader
I20250625 13:52:12.570264 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ace674b2cf3442dab6b85b1232aa8b86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.572866 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace674b2cf3442dab6b85b1232aa8b86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.574057 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_66528a87070643a69c4220dd80f34de0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.574590 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66528a87070643a69c4220dd80f34de0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.575927 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9f776c4576a64684b3f3387da0b9923b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.576673 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f776c4576a64684b3f3387da0b9923b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.589833 1170 tablet_bootstrap.cc:700] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent opening tablet: real 0.525s user 0.000s sys 0.000s
I20250625 13:52:12.590205 1170 tablet_bootstrap.cc:654] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.597044 590 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
I20250625 13:52:12.612262 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
I20250625 13:52:12.617637 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e4c58462d2844a6be928491924b59fb" } modify_external_catalogs: true
I20250625 13:52:12.617834 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e4c58462d2844a6be928491924b59fb" } modify_external_catalogs: true
I20250625 13:52:12.618480 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_688372ad7a204de3aa17a0fcf7c0a34f" } modify_external_catalogs: true
I20250625 13:52:12.618654 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_688372ad7a204de3aa17a0fcf7c0a34f" } modify_external_catalogs: true
I20250625 13:52:12.624205 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_baa360c7bf404bb79b9e32c5d95dc8b5" } modify_external_catalogs: true
I20250625 13:52:12.624243 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40e114808bdd4477ae9ba7f8f92f6f8c" } modify_external_catalogs: true
I20250625 13:52:12.624428 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_baa360c7bf404bb79b9e32c5d95dc8b5" } modify_external_catalogs: true
I20250625 13:52:12.624432 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40e114808bdd4477ae9ba7f8f92f6f8c" } modify_external_catalogs: true
I20250625 13:52:12.626672 2254 ts_tablet_manager.cc:1918] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:12.632583 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
I20250625 13:52:12.634807 2308 tablet.cc:1722] T 5e280bd329764924a6fd42c591888ed0 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:12.642604 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e1cdf5a629142a98879f22c244338eb" } modify_external_catalogs: true
I20250625 13:52:12.642848 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6e1cdf5a629142a98879f22c244338eb" } modify_external_catalogs: true
I20250625 13:52:12.645375 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6340de862fd04c12bb7f85557cfe53d2
I20250625 13:52:12.646761 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 6340de862fd04c12bb7f85557cfe53d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:12.647485 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6340de862fd04c12bb7f85557cfe53d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6340de862fd04c12bb7f85557cfe53d2 already in progress: creating tablet
I20250625 13:52:12.648298 1166 tablet_bootstrap.cc:492] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:12.648445 1166 ts_tablet_manager.cc:1397] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.342s user 0.018s sys 0.026s
I20250625 13:52:12.648458 597 master_service.cc:948] ReplaceTablet: received request to replace tablet ef1f9c03b0ef455cad05b3fe59bdd67c from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:12.649303 1166 raft_consensus.cc:357] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.649381 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8063f1e6d8a0469f8331ae4de006bf77 (DEFAULT_TABLE table=default.Table_7096ca87d4754bec97389b249556e073 [id=eff076950a0e4844a5461755356d35b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.649471 1166 raft_consensus.cc:383] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.649547 1166 raft_consensus.cc:738] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.649771 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8063f1e6d8a0469f8331ae4de006bf77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.649748 1166 consensus_queue.cc:260] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.650195 1166 ts_tablet_manager.cc:1428] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:12.650470 1166 tablet_bootstrap.cc:492] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:12.653141 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d74b5058e9b149069173bc5409364d6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.653498 2254 log.cc:1199] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/063a793e0d7747668d2728fdc2160b5e
W20250625 13:52:12.653678 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d74b5058e9b149069173bc5409364d6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.653925 2254 ts_tablet_manager.cc:1939] T 063a793e0d7747668d2728fdc2160b5e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:12.654232 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a85fa343e30e4e838ab81cd775c298ba
I20250625 13:52:12.654386 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e597f8cd8e5a4ec6be8db57f65c53c65
I20250625 13:52:12.654595 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 063a793e0d7747668d2728fdc2160b5e (table default.Table_9c6d8799477b4081b2652a4df3b13762 [id=3398f6ade26340dcae5b0da2eb58034c]) successfully deleted
I20250625 13:52:12.654997 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f3c75deb64c74f0282efeed827659ec2
I20250625 13:52:12.655841 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a85fa343e30e4e838ab81cd775c298ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.656570 749 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c75deb64c74f0282efeed827659ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.657289 742 tablet_service.cc:1515] Processing DeleteTablet for tablet a85fa343e30e4e838ab81cd775c298ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.657485 754 tablet_service.cc:1515] Processing DeleteTablet for tablet e597f8cd8e5a4ec6be8db57f65c53c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.657513 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet e597f8cd8e5a4ec6be8db57f65c53c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.639995 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 8063f1e6d8a0469f8331ae4de006bf77 (DEFAULT_TABLE table=default.Table_7096ca87d4754bec97389b249556e073 [id=eff076950a0e4844a5461755356d35b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.658145 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dfaba56442c0498da95e6fad126ab86d (table default.Table_f4de7823c10e40e1bdd42b0e0559f7bf [id=7270b5aa9a0148178683a229db5b4b7a]). Expected version 1 got 0
I20250625 13:52:12.658262 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8063f1e6d8a0469f8331ae4de006bf77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.658740 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f3c75deb64c74f0282efeed827659ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3c75deb64c74f0282efeed827659ec2 already in progress: creating tablet
W20250625 13:52:12.658995 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a85fa343e30e4e838ab81cd775c298ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a85fa343e30e4e838ab81cd775c298ba already in progress: creating tablet
W20250625 13:52:12.659175 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e597f8cd8e5a4ec6be8db57f65c53c65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e597f8cd8e5a4ec6be8db57f65c53c65 already in progress: creating tablet
I20250625 13:52:12.651420 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 6340de862fd04c12bb7f85557cfe53d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.651433 1170 tablet_bootstrap.cc:492] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:12.661957 1170 ts_tablet_manager.cc:1397] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.608s user 0.004s sys 0.000s
I20250625 13:52:12.662240 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bd036ec08f774f5393067ce20bb07610
I20250625 13:52:12.662489 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_afac01f597474109bbc2f29c5279b1fa" } new_table_name: "default.Table_2de49834c22b4922b4d0bdd92dd76e3f" modify_external_catalogs: true
I20250625 13:52:12.662786 1170 raft_consensus.cc:357] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:12.662953 1170 raft_consensus.cc:383] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.663010 1170 raft_consensus.cc:738] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.663209 1170 consensus_queue.cc:260] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:12.663585 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ef1f9c03b0ef455cad05b3fe59bdd67c
I20250625 13:52:12.664021 597 catalog_manager.cc:6652] ReplaceTablet: tablet ef1f9c03b0ef455cad05b3fe59bdd67c deleted and replaced by tablet 0bc136dc84de44b49abcdf4100a86137
I20250625 13:52:12.664469 1170 ts_tablet_manager.cc:1428] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:12.665225 1166 tablet_bootstrap.cc:654] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.668051 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3774c3c6876d4c799d85694da32c4f74" } new_table_name: "default.Table_ef2cb3b80b6640c6a511b88a051fa7b4" modify_external_catalogs: true
I20250625 13:52:12.668259 754 tablet_service.cc:1515] Processing DeleteTablet for tablet ef1f9c03b0ef455cad05b3fe59bdd67c with delete_type TABLET_DATA_DELETED (replaced by tablet 0bc136dc84de44b49abcdf4100a86137) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.668375 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3fc03dbe0a19470fbd0fe550159cd232 (DEFAULT_TABLE table=default.Table_ff276efb7beb42dcaf8cb45ca0c0e04a [id=7513471be374411c8f76b82bc69173ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.670775 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d36d7fc8ec7d4bbeb6f5445c8c1b210d (DEFAULT_TABLE table=default.Table_1e94167cff8f4ee8ad52fa7f9e012a35 [id=031831da59e54c3aa1d464d71ad2d91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.671010 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a4a8e4337fea4493bd07b556314e49db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.671164 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 0ba665fbfdb647bc9b4b5bcbff05de94 (DEFAULT_TABLE table=default.Table_da70d46ae8854be99605668d2d0fdf95 [id=3d42f18b1b0049ed9011fdd3f7ec2724]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.671551 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4a8e4337fea4493bd07b556314e49db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.671509 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 15e8a1375f7341559ff9eb2d8180fb83 (DEFAULT_TABLE table=default.Table_957dc0554e3040d38f97ccfa311f4270 [id=2b85ef988fc74560bd3080a5e5bde122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.673162 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ac4439707ad3409595f54dce3c9b810f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.673660 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4439707ad3409595f54dce3c9b810f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.673791 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet dfaba56442c0498da95e6fad126ab86d (table default.Table_f4de7823c10e40e1bdd42b0e0559f7bf [id=7270b5aa9a0148178683a229db5b4b7a]). Expected version 1 got 0
I20250625 13:52:12.674160 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cf5bc408ef7141fa9893bdf60c24d10b" } new_table_name: "default.Table_6d7df5c325b44ea1899f5faa70e7c37d" modify_external_catalogs: true
I20250625 13:52:12.674664 1170 tablet_bootstrap.cc:492] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:12.676220 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc03dbe0a19470fbd0fe550159cd232. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.682188 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet ef1f9c03b0ef455cad05b3fe59bdd67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef1f9c03b0ef455cad05b3fe59bdd67c already in progress: creating tablet
I20250625 13:52:12.682561 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a0e81d4c904248eba858462bcf31c5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:12.684041 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2191b836bb3044c89fee13c869d9e07d consensus state has no leader
I20250625 13:52:12.684095 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0ba665fbfdb647bc9b4b5bcbff05de94 (DEFAULT_TABLE table=default.Table_da70d46ae8854be99605668d2d0fdf95 [id=3d42f18b1b0049ed9011fdd3f7ec2724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.684484 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba665fbfdb647bc9b4b5bcbff05de94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.685226 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet ef1f9c03b0ef455cad05b3fe59bdd67c with delete_type TABLET_DATA_DELETED (replaced by tablet 0bc136dc84de44b49abcdf4100a86137) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.685801 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 466717307dd04d1cbdac9f03229f6a3b (DEFAULT_TABLE table=default.Table_2cc08863cc3d48d99668f2fac71fa727 [id=6040122bcfcc417da47a994e01e19bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.686983 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 3da90cb5c8dd458386ba6bd1d3891ea1 (DEFAULT_TABLE table=default.Table_bbc520f4e79e430b9629ee6fe8682d1a [id=8916c2be3fb24c0bb41bd3299bc7ef0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.687137 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 952fdcc9d1234d31a9c45b6fbcd146bf (DEFAULT_TABLE table=default.Table_6ad4a7572a6941ca8392da904cd328bb [id=fcd8c782d27d4df1ad2be850c767302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.687480 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c75deb64c74f0282efeed827659ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.687636 741 tablet_service.cc:1468] Processing CreateTablet for tablet d36d7fc8ec7d4bbeb6f5445c8c1b210d (DEFAULT_TABLE table=default.Table_1e94167cff8f4ee8ad52fa7f9e012a35 [id=031831da59e54c3aa1d464d71ad2d91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.687731 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet bd036ec08f774f5393067ce20bb07610 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.688037 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36d7fc8ec7d4bbeb6f5445c8c1b210d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.691586 742 tablet_service.cc:1468] Processing CreateTablet for tablet 15e8a1375f7341559ff9eb2d8180fb83 (DEFAULT_TABLE table=default.Table_957dc0554e3040d38f97ccfa311f4270 [id=2b85ef988fc74560bd3080a5e5bde122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.691927 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e8a1375f7341559ff9eb2d8180fb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.692368 1170 tablet_bootstrap.cc:654] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.694984 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_192bf6df15cb499a89d7a61c09717d78" } new_table_name: "default.Table_c40946cbeb1b49f69da130828749450e" modify_external_catalogs: true
I20250625 13:52:12.695479 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3fc03dbe0a19470fbd0fe550159cd232 (DEFAULT_TABLE table=default.Table_ff276efb7beb42dcaf8cb45ca0c0e04a [id=7513471be374411c8f76b82bc69173ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.695899 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc03dbe0a19470fbd0fe550159cd232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.696514 747 tablet_service.cc:1515] Processing DeleteTablet for tablet bd036ec08f774f5393067ce20bb07610 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.538082 1171 ts_tablet_manager.cc:1428] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.207s user 0.003s sys 0.000s
I20250625 13:52:12.698184 1171 tablet_bootstrap.cc:492] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:12.569357 2204 raft_consensus.cc:491] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.699821 2204 raft_consensus.cc:513] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:12.607550 2303 tablet.cc:1722] T 5e280bd329764924a6fd42c591888ed0 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250625 13:52:12.616176 849 consensus_peers.cc:487] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:12.700848 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 84707.3us
I20250625 13:52:12.704527 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32143e2497ee4d1b8c2e7cbfc6160ef4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:12.704705 773 raft_consensus.cc:2466] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:12.705605 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 26530e4206b34be29c7e822aa4ed1c93 (DEFAULT_TABLE table=default.Table_f7ba037477124796ab2a66a351968c19 [id=2fe7121baaa140ca8670b9e39e572fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.706403 1021 tablet_service.cc:1468] Processing CreateTablet for tablet d7f68a0c4bb341a9a90cc2be0f78daca (DEFAULT_TABLE table=default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179 [id=617f5dfcd64d408ea32281faf0fa6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.706516 1018 tablet_service.cc:1468] Processing CreateTablet for tablet ca97e8f38f534c9aad519f11cbf85283 (DEFAULT_TABLE table=default.Table_800985c199634630bea2fe2713086182 [id=cc96052bf1344ff293e4167564954d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.647377 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 6340de862fd04c12bb7f85557cfe53d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.657742 891 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c75deb64c74f0282efeed827659ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.706810 1013 tablet_service.cc:1468] Processing CreateTablet for tablet a89db9678dd444d5a4df85baccd62044 (DEFAULT_TABLE table=default.Table_6bb83d62643e419cbe5a1ac4c61fea12 [id=4427063cb5bd4e9fb83c964bafff214b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.657900 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a85fa343e30e4e838ab81cd775c298ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.661710 895 tablet_service.cc:1515] Processing DeleteTablet for tablet e597f8cd8e5a4ec6be8db57f65c53c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.665439 882 tablet_service.cc:1515] Processing DeleteTablet for tablet ef1f9c03b0ef455cad05b3fe59bdd67c with delete_type TABLET_DATA_DELETED (replaced by tablet 0bc136dc84de44b49abcdf4100a86137) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.667932 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8063f1e6d8a0469f8331ae4de006bf77 (DEFAULT_TABLE table=default.Table_7096ca87d4754bec97389b249556e073 [id=eff076950a0e4844a5461755356d35b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.669857 885 tablet_service.cc:1468] Processing CreateTablet for tablet 3fc03dbe0a19470fbd0fe550159cd232 (DEFAULT_TABLE table=default.Table_ff276efb7beb42dcaf8cb45ca0c0e04a [id=7513471be374411c8f76b82bc69173ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.670167 877 tablet_service.cc:1468] Processing CreateTablet for tablet 15e8a1375f7341559ff9eb2d8180fb83 (DEFAULT_TABLE table=default.Table_957dc0554e3040d38f97ccfa311f4270 [id=2b85ef988fc74560bd3080a5e5bde122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.670440 890 tablet_service.cc:1468] Processing CreateTablet for tablet 0ba665fbfdb647bc9b4b5bcbff05de94 (DEFAULT_TABLE table=default.Table_da70d46ae8854be99605668d2d0fdf95 [id=3d42f18b1b0049ed9011fdd3f7ec2724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.707580 851 leader_election.cc:304] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:12.685056 881 tablet_service.cc:1515] Processing DeleteTablet for tablet bd036ec08f774f5393067ce20bb07610 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.708158 1168 log.cc:1199] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/9c1deb52366c404cbdc08076d4bcb3d1
W20250625 13:52:12.715287 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bd036ec08f774f5393067ce20bb07610 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd036ec08f774f5393067ce20bb07610 already in progress: creating tablet
I20250625 13:52:12.716830 743 tablet_service.cc:1468] Processing CreateTablet for tablet d7f68a0c4bb341a9a90cc2be0f78daca (DEFAULT_TABLE table=default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179 [id=617f5dfcd64d408ea32281faf0fa6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.716807 747 tablet_service.cc:1468] Processing CreateTablet for tablet a89db9678dd444d5a4df85baccd62044 (DEFAULT_TABLE table=default.Table_6bb83d62643e419cbe5a1ac4c61fea12 [id=4427063cb5bd4e9fb83c964bafff214b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.717216 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f68a0c4bb341a9a90cc2be0f78daca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.717406 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89db9678dd444d5a4df85baccd62044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.720937 755 tablet_service.cc:1468] Processing CreateTablet for tablet 26530e4206b34be29c7e822aa4ed1c93 (DEFAULT_TABLE table=default.Table_f7ba037477124796ab2a66a351968c19 [id=2fe7121baaa140ca8670b9e39e572fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.721302 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26530e4206b34be29c7e822aa4ed1c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.724222 2192 raft_consensus.cc:491] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.724365 2192 raft_consensus.cc:513] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.725098 2192 leader_election.cc:290] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:12.725741 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c12f55ed5b4425aea1bf45cdc0658e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:12.720996 750 tablet_service.cc:1468] Processing CreateTablet for tablet ca97e8f38f534c9aad519f11cbf85283 (DEFAULT_TABLE table=default.Table_800985c199634630bea2fe2713086182 [id=cc96052bf1344ff293e4167564954d88]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.726382 849 leader_election.cc:343] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.726469 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca97e8f38f534c9aad519f11cbf85283. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.727208 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c12f55ed5b4425aea1bf45cdc0658e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:12.727413 773 raft_consensus.cc:2466] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:12.727541 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36d7fc8ec7d4bbeb6f5445c8c1b210d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.727813 851 leader_election.cc:304] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:12.728206 2192 raft_consensus.cc:2802] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:12.728276 2192 raft_consensus.cc:491] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:12.728330 2192 raft_consensus.cc:3058] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:12.731837 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26530e4206b34be29c7e822aa4ed1c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.735081 2204 leader_election.cc:290] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:12.735663 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32143e2497ee4d1b8c2e7cbfc6160ef4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:12.735903 881 tablet_service.cc:1468] Processing CreateTablet for tablet d36d7fc8ec7d4bbeb6f5445c8c1b210d (DEFAULT_TABLE table=default.Table_1e94167cff8f4ee8ad52fa7f9e012a35 [id=031831da59e54c3aa1d464d71ad2d91a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.736214 849 leader_election.cc:343] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.737108 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8063f1e6d8a0469f8331ae4de006bf77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.737906 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc03dbe0a19470fbd0fe550159cd232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.738485 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e8a1375f7341559ff9eb2d8180fb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.738692 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e8a1375f7341559ff9eb2d8180fb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.740785 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba665fbfdb647bc9b4b5bcbff05de94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.742726 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0af46ccac2054224a7c0186bbc940d21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.743433 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0af46ccac2054224a7c0186bbc940d21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.748345 749 tablet_service.cc:1468] Processing CreateTablet for tablet 3da90cb5c8dd458386ba6bd1d3891ea1 (DEFAULT_TABLE table=default.Table_bbc520f4e79e430b9629ee6fe8682d1a [id=8916c2be3fb24c0bb41bd3299bc7ef0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.749058 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da90cb5c8dd458386ba6bd1d3891ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.753472 742 tablet_service.cc:1468] Processing CreateTablet for tablet 952fdcc9d1234d31a9c45b6fbcd146bf (DEFAULT_TABLE table=default.Table_6ad4a7572a6941ca8392da904cd328bb [id=fcd8c782d27d4df1ad2be850c767302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.753865 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 952fdcc9d1234d31a9c45b6fbcd146bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:12.754030 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7c6a325d40c4f7ba2f342e39b9437e3 consensus state has no leader
W20250625 13:52:12.755276 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 170ffbff76af447a83d663fa8688b7a2 consensus state has no leader
I20250625 13:52:12.757856 2254 tablet_replica.cc:331] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:12.758092 2254 raft_consensus.cc:2241] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:12.758160 741 tablet_service.cc:1468] Processing CreateTablet for tablet 466717307dd04d1cbdac9f03229f6a3b (DEFAULT_TABLE table=default.Table_2cc08863cc3d48d99668f2fac71fa727 [id=6040122bcfcc417da47a994e01e19bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.758505 2254 raft_consensus.cc:2270] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.758539 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 466717307dd04d1cbdac9f03229f6a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.759392 2254 ts_tablet_manager.cc:1905] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.763404 1168 ts_tablet_manager.cc:1939] T 9c1deb52366c404cbdc08076d4bcb3d1 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:12.763789 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 812) took 4254 ms. Trace:
I20250625 13:52:12.763939 1168 rpcz_store.cc:276] 0625 13:52:08.509135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.509189 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:12.039749 (+3530560us) ts_tablet_manager.cc:1145] Deleting tablet 9c1deb52366c404cbdc08076d4bcb3d1
0625 13:52:12.039764 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:12.534393 (+494629us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:12.537618 (+ 3225us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:12.763773 (+226155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3530341}
I20250625 13:52:12.764475 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 816) took 4197 ms. Trace:
I20250625 13:52:12.766575 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 952fdcc9d1234d31a9c45b6fbcd146bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.767674 1170 tablet_bootstrap.cc:492] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:12.767797 1170 ts_tablet_manager.cc:1397] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.000s
I20250625 13:52:12.768568 1170 raft_consensus.cc:357] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.768718 1170 raft_consensus.cc:383] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.768769 1170 raft_consensus.cc:738] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.768947 1170 consensus_queue.cc:260] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.769523 1170 ts_tablet_manager.cc:1428] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:12.769783 1170 tablet_bootstrap.cc:492] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:12.770911 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f68a0c4bb341a9a90cc2be0f78daca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.771164 1170 tablet_bootstrap.cc:654] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.775245 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89db9678dd444d5a4df85baccd62044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.781773 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca97e8f38f534c9aad519f11cbf85283. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.784080 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da90cb5c8dd458386ba6bd1d3891ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.784044 895 tablet_service.cc:1468] Processing CreateTablet for tablet 466717307dd04d1cbdac9f03229f6a3b (DEFAULT_TABLE table=default.Table_2cc08863cc3d48d99668f2fac71fa727 [id=6040122bcfcc417da47a994e01e19bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.785144 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 466717307dd04d1cbdac9f03229f6a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.787364 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36d7fc8ec7d4bbeb6f5445c8c1b210d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.787572 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba665fbfdb647bc9b4b5bcbff05de94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.793253 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 466717307dd04d1cbdac9f03229f6a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.793728 2313 raft_consensus.cc:491] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.795398 2313 raft_consensus.cc:513] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.764618 1168 rpcz_store.cc:276] 0625 13:52:08.566737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.566971 (+ 234us) service_pool.cc:225] Handling call
0625 13:52:12.764345 (+4197374us) ts_tablet_manager.cc:1145] Deleting tablet 1d9bd351536c4558b189cebf8b793343
0625 13:52:12.764355 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:12.764460 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4197209}
I20250625 13:52:12.796252 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 818) took 4214 ms. Trace:
I20250625 13:52:12.796343 1168 rpcz_store.cc:276] 0625 13:52:08.581710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.581819 (+ 109us) service_pool.cc:225] Handling call
0625 13:52:12.796120 (+4214301us) ts_tablet_manager.cc:1145] Deleting tablet 9f3212d0b8de440e82156323ecb40a7d
0625 13:52:12.796134 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:12.796234 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4214130}
I20250625 13:52:12.796649 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 821) took 4181 ms. Trace:
I20250625 13:52:12.796711 1168 rpcz_store.cc:276] 0625 13:52:08.615055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.615451 (+ 396us) service_pool.cc:225] Handling call
0625 13:52:12.796567 (+4181116us) ts_tablet_manager.cc:1145] Deleting tablet c7ad5e90188847f38894fc3c1f8f2693
0625 13:52:12.796575 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:12.796639 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4180867}
I20250625 13:52:12.797453 882 tablet_service.cc:1468] Processing CreateTablet for tablet 952fdcc9d1234d31a9c45b6fbcd146bf (DEFAULT_TABLE table=default.Table_6ad4a7572a6941ca8392da904cd328bb [id=fcd8c782d27d4df1ad2be850c767302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.797590 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:12.798156 711 leader_election.cc:343] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.798362 891 tablet_service.cc:1468] Processing CreateTablet for tablet a89db9678dd444d5a4df85baccd62044 (DEFAULT_TABLE table=default.Table_6bb83d62643e419cbe5a1ac4c61fea12 [id=4427063cb5bd4e9fb83c964bafff214b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.798697 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89db9678dd444d5a4df85baccd62044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.798794 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f5c726e169ca41b6a7092eef2e923eb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.799019 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3da90cb5c8dd458386ba6bd1d3891ea1 (DEFAULT_TABLE table=default.Table_bbc520f4e79e430b9629ee6fe8682d1a [id=8916c2be3fb24c0bb41bd3299bc7ef0b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.799373 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5c726e169ca41b6a7092eef2e923eb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.800698 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_021ec65b75b74e9db71be9078b4a1067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.801174 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_021ec65b75b74e9db71be9078b4a1067 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.801627 2271 raft_consensus.cc:2802] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:12.801716 2271 raft_consensus.cc:491] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:12.801771 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 952fdcc9d1234d31a9c45b6fbcd146bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.801787 2271 raft_consensus.cc:3058] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:12.802634 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0837b1ebf17d4ba4a4b0479fe815f9b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.803189 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0837b1ebf17d4ba4a4b0479fe815f9b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.803786 2192 raft_consensus.cc:513] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:12.804996 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4616015b587a459abfb5403b690d23cf consensus state has no leader
I20250625 13:52:12.805641 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 822) took 4178 ms. Trace:
I20250625 13:52:12.805720 1168 rpcz_store.cc:276] 0625 13:52:08.627019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.627121 (+ 102us) service_pool.cc:225] Handling call
0625 13:52:12.796881 (+4169760us) ts_tablet_manager.cc:1145] Deleting tablet 81a359c798024b2a99aa3b307cc1967f
0625 13:52:12.796888 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:12.805619 (+ 8731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4166694}
I20250625 13:52:12.806064 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 811) took 4297 ms. Trace:
I20250625 13:52:12.806123 1168 rpcz_store.cc:276] 0625 13:52:08.508620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.683329 (+174709us) service_pool.cc:225] Handling call
0625 13:52:12.805961 (+4122632us) ts_tablet_manager.cc:1145] Deleting tablet dc1c9cea3d5a405a958c6c7585dbef39
0625 13:52:12.805971 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:12.806052 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4122411}
I20250625 13:52:12.806350 1168 tablet_replica.cc:331] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:12.806535 1168 raft_consensus.cc:2241] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.806667 1168 raft_consensus.cc:2270] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.807160 1168 ts_tablet_manager.cc:1905] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.809370 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3d5b64d4b9ce4cd3a8b4625159a26e70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.809937 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d5b64d4b9ce4cd3a8b4625159a26e70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:12.810817 2162 log.cc:927] Time spent T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Append to log took a long time: real 0.115s user 0.002s sys 0.000s
I20250625 13:52:12.810994 588 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a] alter complete (version 1)
I20250625 13:52:12.814940 1009 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 162) took 1129 ms. Trace:
I20250625 13:52:12.815035 1009 rpcz_store.cc:276] 0625 13:52:11.685511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.699402 (+ 13891us) service_pool.cc:225] Handling call
0625 13:52:12.814916 (+1115514us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:52:12.816520 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fb12128ffae94ceabd5af53e908db5d2" } new_table_name: "default.Table_469944f606314307b97ad3e50d27ad30" modify_external_catalogs: true
I20250625 13:52:12.817834 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_77d42655ef0349128a8518e3892d4532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.818318 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77d42655ef0349128a8518e3892d4532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.819779 2254 ts_tablet_manager.cc:1918] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:12.819885 2254 log.cc:1199] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/be3313ec5dfb41b68662d3ff4c8c94f9
I20250625 13:52:12.820245 2254 ts_tablet_manager.cc:1939] T be3313ec5dfb41b68662d3ff4c8c94f9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:12.821215 2254 tablet_replica.cc:331] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:12.821386 2254 raft_consensus.cc:2241] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.821530 2254 raft_consensus.cc:2270] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.822013 2254 ts_tablet_manager.cc:1905] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.825019 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cf58706a138644a49767bbb2201a120e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.826042 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf58706a138644a49767bbb2201a120e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.834065 594 master_service.cc:948] ReplaceTablet: received request to replace tablet d2972d4366c143498f746fd62a039acd from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:12.836133 880 tablet_service.cc:1468] Processing CreateTablet for tablet 26530e4206b34be29c7e822aa4ed1c93 (DEFAULT_TABLE table=default.Table_f7ba037477124796ab2a66a351968c19 [id=2fe7121baaa140ca8670b9e39e572fea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.836557 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26530e4206b34be29c7e822aa4ed1c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.837702 1170 tablet_bootstrap.cc:492] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:12.837826 1170 ts_tablet_manager.cc:1397] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.000s
I20250625 13:52:12.838462 1170 raft_consensus.cc:357] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:12.840042 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da90cb5c8dd458386ba6bd1d3891ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.842360 2192 leader_election.cc:290] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:12.843775 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 9c1deb52366c404cbdc08076d4bcb3d1 (table default.Table_b2b0087eeea54823809b847648141d81 [id=253f3f5c5b8d4051800dc823b4ebaec9]) successfully deleted
W20250625 13:52:12.844060 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1d9bd351536c4558b189cebf8b793343 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d9bd351536c4558b189cebf8b793343 already in progress: creating tablet
W20250625 13:52:12.844259 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9f3212d0b8de440e82156323ecb40a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f3212d0b8de440e82156323ecb40a7d already in progress: creating tablet
W20250625 13:52:12.844805 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c7ad5e90188847f38894fc3c1f8f2693 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7ad5e90188847f38894fc3c1f8f2693 already in progress: creating tablet
W20250625 13:52:12.845086 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 81a359c798024b2a99aa3b307cc1967f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81a359c798024b2a99aa3b307cc1967f already in progress: creating tablet
W20250625 13:52:12.845316 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet dc1c9cea3d5a405a958c6c7585dbef39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc1c9cea3d5a405a958c6c7585dbef39 already in progress: creating tablet
I20250625 13:52:12.845654 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet be3313ec5dfb41b68662d3ff4c8c94f9 (table default.Table_fb311c42df024421b325596e0cb7a778 [id=7b67603755e7428496524ee716b435e9]) successfully deleted
W20250625 13:52:12.845919 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2593db823e974f1eb653470d5a23338f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2593db823e974f1eb653470d5a23338f already in progress: creating tablet
W20250625 13:52:12.846084 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c9d010744e88406a828c6ba7b76eda3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9d010744e88406a828c6ba7b76eda3e already in progress: creating tablet
W20250625 13:52:12.846258 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 54ea4c4d7756433d85fe9b816b4816ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54ea4c4d7756433d85fe9b816b4816ae already in progress: creating tablet
I20250625 13:52:12.843132 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c12f55ed5b4425aea1bf45cdc0658e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:12.847121 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_73b8d45e1da448d481ef5bbb2191c6cd" } modify_external_catalogs: true
W20250625 13:52:12.847277 849 leader_election.cc:343] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.847335 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_73b8d45e1da448d481ef5bbb2191c6cd" } modify_external_catalogs: true
I20250625 13:52:12.847536 1170 raft_consensus.cc:383] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.847739 1170 raft_consensus.cc:738] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.848348 1170 consensus_queue.cc:260] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:12.851002 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1deb52366c404cbdc08076d4bcb3d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.856736 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aca9eb55cf01477092c2c2288bdab60b" } modify_external_catalogs: true
I20250625 13:52:12.856959 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aca9eb55cf01477092c2c2288bdab60b" } modify_external_catalogs: true
I20250625 13:52:12.857525 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ac6acb4e2c0e4001927ffb4230ad2ecf" } modify_external_catalogs: true
I20250625 13:52:12.857677 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ac6acb4e2c0e4001927ffb4230ad2ecf" } modify_external_catalogs: true
I20250625 13:52:12.861405 1166 tablet_bootstrap.cc:492] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:12.861553 1166 ts_tablet_manager.cc:1397] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.211s user 0.002s sys 0.006s
I20250625 13:52:12.863114 1166 raft_consensus.cc:357] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.863373 1166 raft_consensus.cc:383] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.863443 1166 raft_consensus.cc:738] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.863636 1166 consensus_queue.cc:260] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.864086 1166 ts_tablet_manager.cc:1428] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:12.864866 1170 ts_tablet_manager.cc:1428] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250625 13:52:12.865451 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_736e3a7bd9a44cfda965390534b95893" } modify_external_catalogs: true
I20250625 13:52:12.865664 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_736e3a7bd9a44cfda965390534b95893" } modify_external_catalogs: true
I20250625 13:52:12.868253 2313 leader_election.cc:290] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:12.870882 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c12f55ed5b4425aea1bf45cdc0658e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:12.871045 773 raft_consensus.cc:3058] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:12.872262 2210 consensus_queue.cc:1035] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:12.874159 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2cc08863cc3d48d99668f2fac71fa727" } modify_external_catalogs: true
I20250625 13:52:12.874374 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2cc08863cc3d48d99668f2fac71fa727" } modify_external_catalogs: true
W20250625 13:52:12.874507 849 consensus_peers.cc:487] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b129117db064a15b27e82439637cf22. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:12.875028 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22ee97e6804a4b9fb564ed6f226bc1e0 consensus state has no leader
I20250625 13:52:12.877097 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d41a791307e142c3b46cd3fda9482764"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.880247 1166 tablet_bootstrap.cc:492] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:12.880734 1171 tablet_bootstrap.cc:654] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.881865 883 tablet_service.cc:1468] Processing CreateTablet for tablet ca97e8f38f534c9aad519f11cbf85283 (DEFAULT_TABLE table=default.Table_800985c199634630bea2fe2713086182 [id=cc96052bf1344ff293e4167564954d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.882115 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.882341 1165 tablet_replica.cc:331] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:12.882499 1165 raft_consensus.cc:2241] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:12.882647 1165 raft_consensus.cc:2270] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:12.885715 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_39575a8dc48640dbae773650f2417b76" } new_table_name: "default.Table_232257f7ad2d4eb689c206dfde0510fd" modify_external_catalogs: true
I20250625 13:52:12.888375 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca97e8f38f534c9aad519f11cbf85283. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:12.889869 1165 ts_tablet_manager.cc:1905] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:12.896477 1166 tablet_bootstrap.cc:654] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.896689 773 raft_consensus.cc:2466] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:12.898449 851 leader_election.cc:304] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:12.899117 2192 raft_consensus.cc:2802] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:12.899264 2192 raft_consensus.cc:695] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:12.899585 2192 consensus_queue.cc:237] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.904579 1170 tablet_bootstrap.cc:492] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:12.906857 596 master_service.cc:948] ReplaceTablet: received request to replace tablet dfaba56442c0498da95e6fad126ab86d from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:12.907503 2313 raft_consensus.cc:491] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:12.915256 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:12.918506 2313 raft_consensus.cc:513] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.920145 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:12.921363 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfaba56442c0498da95e6fad126ab86d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:12.921695 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d41a791307e142c3b46cd3fda9482764 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:12.921967 711 leader_election.cc:343] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.923729 1170 tablet_bootstrap.cc:654] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:12.926172 2313 leader_election.cc:290] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:12.936084 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 466717307dd04d1cbdac9f03229f6a3b
I20250625 13:52:12.840788 896 tablet_service.cc:1468] Processing CreateTablet for tablet d7f68a0c4bb341a9a90cc2be0f78daca (DEFAULT_TABLE table=default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179 [id=617f5dfcd64d408ea32281faf0fa6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.939342 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 466717307dd04d1cbdac9f03229f6a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.941442 2308 tablet.cc:1722] T 9267df231f354c578c38f452418e523c P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:12.945910 2266 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:12.947243 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 47fa53b050494194947585dbca53f335 (DEFAULT_TABLE table=default.Table_ace674b2cf3442dab6b85b1232aa8b86 [id=def23f254c174d8698c3471d23dcc50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.947878 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 710ad7e9c9664da9ab53f13455ba1256 (DEFAULT_TABLE table=default.Table_b033e3575af14620a4ffd13d75b56bce [id=e09489ddd6414836b523d5991df20911]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.948410 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 20841d3424fb4c48b4de4ed7b2917abe (DEFAULT_TABLE table=default.Table_ef5aa5771b2b412ca7ebce55f40b6eed [id=aad479a261524bfc82061ca28eba23b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.948855 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc (DEFAULT_TABLE table=default.Table_e2e20a31ea3544cf974a8487d18364e7 [id=a267d159ec1f4d4e9224d99b94146c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.949232 755 tablet_service.cc:1468] Processing CreateTablet for tablet 391b412dbb9e420c90aa79473bbbe8d7 (DEFAULT_TABLE table=default.Table_17343b483b8e4268a6a3cca32bbcb186 [id=e5bbd1cc623a47ecb90443b1f69c0af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.949344 754 tablet_service.cc:1468] Processing CreateTablet for tablet 20841d3424fb4c48b4de4ed7b2917abe (DEFAULT_TABLE table=default.Table_ef5aa5771b2b412ca7ebce55f40b6eed [id=aad479a261524bfc82061ca28eba23b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.949625 750 tablet_service.cc:1468] Processing CreateTablet for tablet 5e7bf419440645279046b030e7554893 (DEFAULT_TABLE table=default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0 [id=63ad0995448746079f9b85d82eaae0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.949775 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc (DEFAULT_TABLE table=default.Table_e2e20a31ea3544cf974a8487d18364e7 [id=a267d159ec1f4d4e9224d99b94146c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.950316 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 391b412dbb9e420c90aa79473bbbe8d7 (DEFAULT_TABLE table=default.Table_17343b483b8e4268a6a3cca32bbcb186 [id=e5bbd1cc623a47ecb90443b1f69c0af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.950778 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 5e7bf419440645279046b030e7554893 (DEFAULT_TABLE table=default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0 [id=63ad0995448746079f9b85d82eaae0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.951248 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 8faa58acd09649eab1a622e8c1ddb224 (DEFAULT_TABLE table=default.Table_9f776c4576a64684b3f3387da0b9923b [id=34ca94f083ca404bbd38efafeca91d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.951756 741 tablet_service.cc:1468] Processing CreateTablet for tablet 710ad7e9c9664da9ab53f13455ba1256 (DEFAULT_TABLE table=default.Table_b033e3575af14620a4ffd13d75b56bce [id=e09489ddd6414836b523d5991df20911]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.952486 742 tablet_service.cc:1468] Processing CreateTablet for tablet 47fa53b050494194947585dbca53f335 (DEFAULT_TABLE table=default.Table_ace674b2cf3442dab6b85b1232aa8b86 [id=def23f254c174d8698c3471d23dcc50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.952559 2322 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:12.939126 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfaba56442c0498da95e6fad126ab86d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:12.953238 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 466717307dd04d1cbdac9f03229f6a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:12.953956 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:12.956693 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff091cba95574357ba76fc2e306f3666" } new_table_name: "default.Table_001eda8fd8bf4b5b884fc36818b52fe0" modify_external_catalogs: true
I20250625 13:52:12.959865 890 tablet_service.cc:1468] Processing CreateTablet for tablet 710ad7e9c9664da9ab53f13455ba1256 (DEFAULT_TABLE table=default.Table_b033e3575af14620a4ffd13d75b56bce [id=e09489ddd6414836b523d5991df20911]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.961020 888 tablet_service.cc:1468] Processing CreateTablet for tablet 20841d3424fb4c48b4de4ed7b2917abe (DEFAULT_TABLE table=default.Table_ef5aa5771b2b412ca7ebce55f40b6eed [id=aad479a261524bfc82061ca28eba23b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.965452 878 tablet_service.cc:1468] Processing CreateTablet for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc (DEFAULT_TABLE table=default.Table_e2e20a31ea3544cf974a8487d18364e7 [id=a267d159ec1f4d4e9224d99b94146c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.965780 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5e7bf419440645279046b030e7554893 (DEFAULT_TABLE table=default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0 [id=63ad0995448746079f9b85d82eaae0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.966631 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8faa58acd09649eab1a622e8c1ddb224 (DEFAULT_TABLE table=default.Table_9f776c4576a64684b3f3387da0b9923b [id=34ca94f083ca404bbd38efafeca91d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.966882 746 tablet_service.cc:1468] Processing CreateTablet for tablet c450ae4c434e4c6fa73d06c184b2bdc7 (DEFAULT_TABLE table=default.Table_66528a87070643a69c4220dd80f34de0 [id=ecf37b617fd64f7f8c8613acfef11949]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.966878 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cea4a53bfa854372839e62197285c18c" } new_table_name: "default.Table_c95698028b9241c880fe126c21813367" modify_external_catalogs: true
I20250625 13:52:12.967535 745 tablet_service.cc:1468] Processing CreateTablet for tablet 3d8ce12926e044ee95291a5e9168277a (DEFAULT_TABLE table=default.Table_d74b5058e9b149069173bc5409364d6c [id=17cc7336c1144e4dbe7fe181b32646f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.966663 744 tablet_service.cc:1468] Processing CreateTablet for tablet 410e7c1ea44c42429a02bac3b3dd8b34 (DEFAULT_TABLE table=default.Table_f5cca2b6860a4245a3c3d183b0143909 [id=f662aa1bdd2848198ae3377e349848da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.967208 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0bc136dc84de44b49abcdf4100a86137 (DEFAULT_TABLE table=default.Table_b88679cb7cc74587927204b82e5915f4 [id=8a73adb9e4774a7fb5a53cd8896e2d16]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:12.968756 711 leader_election.cc:343] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.968905 711 leader_election.cc:304] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:12.969537 2315 raft_consensus.cc:2747] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:12.969527 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0bc136dc84de44b49abcdf4100a86137 (DEFAULT_TABLE table=default.Table_b88679cb7cc74587927204b82e5915f4 [id=8a73adb9e4774a7fb5a53cd8896e2d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.970142 1027 tablet_service.cc:1468] Processing CreateTablet for tablet c450ae4c434e4c6fa73d06c184b2bdc7 (DEFAULT_TABLE table=default.Table_66528a87070643a69c4220dd80f34de0 [id=ecf37b617fd64f7f8c8613acfef11949]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.970194 884 tablet_service.cc:1468] Processing CreateTablet for tablet 391b412dbb9e420c90aa79473bbbe8d7 (DEFAULT_TABLE table=default.Table_17343b483b8e4268a6a3cca32bbcb186 [id=e5bbd1cc623a47ecb90443b1f69c0af9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.970499 1012 tablet_service.cc:1468] Processing CreateTablet for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 (DEFAULT_TABLE table=default.Table_78d7cfce8bd9428c8b6f47e48fc1ddb6 [id=793cdbd3a05842b08926f2d6d0434858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.971578 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a1144201285a4e62b6eedd9332730eb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:12.972121 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1144201285a4e62b6eedd9332730eb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:12.973321 711 leader_election.cc:343] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:12.973338 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 466717307dd04d1cbdac9f03229f6a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.973467 711 leader_election.cc:304] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:12.973692 2315 raft_consensus.cc:2747] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:12.970144 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 410e7c1ea44c42429a02bac3b3dd8b34 (DEFAULT_TABLE table=default.Table_f5cca2b6860a4245a3c3d183b0143909 [id=f662aa1bdd2848198ae3377e349848da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.969527 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3d8ce12926e044ee95291a5e9168277a (DEFAULT_TABLE table=default.Table_d74b5058e9b149069173bc5409364d6c [id=17cc7336c1144e4dbe7fe181b32646f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.974480 1170 tablet_bootstrap.cc:492] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:12.974251 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_167a0a50523949028824975d052ed256"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:12.975739 1170 ts_tablet_manager.cc:1397] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
W20250625 13:52:12.976430 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_167a0a50523949028824975d052ed256 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:12.980770 752 tablet_service.cc:1468] Processing CreateTablet for tablet f47980e0c9ad487b8113f13458e175ec (DEFAULT_TABLE table=default.Table_fec914ebb830498da551091c86a181cd [id=0fce93fe153745e6826dc4aec9b00186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.981144 1017 tablet_service.cc:1468] Processing CreateTablet for tablet f47980e0c9ad487b8113f13458e175ec (DEFAULT_TABLE table=default.Table_fec914ebb830498da551091c86a181cd [id=0fce93fe153745e6826dc4aec9b00186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.981681 749 tablet_service.cc:1468] Processing CreateTablet for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 (DEFAULT_TABLE table=default.Table_78d7cfce8bd9428c8b6f47e48fc1ddb6 [id=793cdbd3a05842b08926f2d6d0434858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.982638 887 tablet_service.cc:1468] Processing CreateTablet for tablet 8faa58acd09649eab1a622e8c1ddb224 (DEFAULT_TABLE table=default.Table_9f776c4576a64684b3f3387da0b9923b [id=34ca94f083ca404bbd38efafeca91d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.982956 886 tablet_service.cc:1468] Processing CreateTablet for tablet 0bc136dc84de44b49abcdf4100a86137 (DEFAULT_TABLE table=default.Table_b88679cb7cc74587927204b82e5915f4 [id=8a73adb9e4774a7fb5a53cd8896e2d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.983237 892 tablet_service.cc:1468] Processing CreateTablet for tablet 3d8ce12926e044ee95291a5e9168277a (DEFAULT_TABLE table=default.Table_d74b5058e9b149069173bc5409364d6c [id=17cc7336c1144e4dbe7fe181b32646f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.983533 894 tablet_service.cc:1468] Processing CreateTablet for tablet 47fa53b050494194947585dbca53f335 (DEFAULT_TABLE table=default.Table_ace674b2cf3442dab6b85b1232aa8b86 [id=def23f254c174d8698c3471d23dcc50e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:12.981424 1170 raft_consensus.cc:357] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.987807 2303 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:12.986536 1170 raft_consensus.cc:383] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:12.988443 1170 raft_consensus.cc:738] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:12.990978 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 909f4bb91e784b29acbbc9763e35ab03
I20250625 13:52:12.990013 1170 consensus_queue.cc:260] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:12.991745 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9df1f195befe4aed999b41f770547fde
I20250625 13:52:12.991998 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 909f4bb91e784b29acbbc9763e35ab03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.992682 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:12.994207 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 9df1f195befe4aed999b41f770547fde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.994640 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a1263945be894618a7f9946e7ca50efd
I20250625 13:52:12.994858 1170 ts_tablet_manager.cc:1428] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.019s user 0.000s sys 0.003s
I20250625 13:52:12.995502 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet dfaba56442c0498da95e6fad126ab86d
I20250625 13:52:12.995468 758 tablet_service.cc:1515] Processing DeleteTablet for tablet a1263945be894618a7f9946e7ca50efd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:12.995776 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet fd7a6587aa674a0eaaa107e2a2f673a3
W20250625 13:52:12.996361 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34fe02d7b37f4d99bbae082943223e35 consensus state has no leader
I20250625 13:52:12.996765 596 catalog_manager.cc:6652] ReplaceTablet: tablet dfaba56442c0498da95e6fad126ab86d deleted and replaced by tablet ee09a3942acc41acb4f7c0d5452bf268
W20250625 13:52:12.997046 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3e2076007ad48acaef3aa5357299b2e consensus state has no leader
I20250625 13:52:12.997179 1170 tablet_bootstrap.cc:492] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:12.998025 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 909f4bb91e784b29acbbc9763e35ab03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.998346 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 9df1f195befe4aed999b41f770547fde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.999392 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet dfaba56442c0498da95e6fad126ab86d with delete_type TABLET_DATA_DELETED (replaced by tablet ee09a3942acc41acb4f7c0d5452bf268) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.999588 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet fd7a6587aa674a0eaaa107e2a2f673a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:12.999734 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet a1263945be894618a7f9946e7ca50efd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.000654 2271 raft_consensus.cc:513] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.001305 2271 leader_election.cc:290] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:13.002296 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32143e2497ee4d1b8c2e7cbfc6160ef4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:13.003134 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet df1ed922750f45eab02d1fc0c26af75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.005723 891 tablet_service.cc:1468] Processing CreateTablet for tablet 410e7c1ea44c42429a02bac3b3dd8b34 (DEFAULT_TABLE table=default.Table_f5cca2b6860a4245a3c3d183b0143909 [id=f662aa1bdd2848198ae3377e349848da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.010377 877 tablet_service.cc:1468] Processing CreateTablet for tablet c450ae4c434e4c6fa73d06c184b2bdc7 (DEFAULT_TABLE table=default.Table_66528a87070643a69c4220dd80f34de0 [id=ecf37b617fd64f7f8c8613acfef11949]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:13.010953 849 leader_election.cc:343] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.011669 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32143e2497ee4d1b8c2e7cbfc6160ef4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:13.011857 774 raft_consensus.cc:3058] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:13.011835 1170 tablet_bootstrap.cc:654] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.017228 895 tablet_service.cc:1468] Processing CreateTablet for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 (DEFAULT_TABLE table=default.Table_78d7cfce8bd9428c8b6f47e48fc1ddb6 [id=793cdbd3a05842b08926f2d6d0434858]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.020370 758 tablet_service.cc:1515] Processing DeleteTablet for tablet fd7a6587aa674a0eaaa107e2a2f673a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.020622 753 tablet_service.cc:1515] Processing DeleteTablet for tablet dfaba56442c0498da95e6fad126ab86d with delete_type TABLET_DATA_DELETED (replaced by tablet ee09a3942acc41acb4f7c0d5452bf268) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.021361 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 2404e712e18146b68757bd77362c1358 with delete_type TABLET_DATA_DELETED (replaced by tablet 68831b9ad1c14f5e9c954cbd06094896) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.022068 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8faa58acd09649eab1a622e8c1ddb224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.023468 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20841d3424fb4c48b4de4ed7b2917abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.025676 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410e7c1ea44c42429a02bac3b3dd8b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.026167 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0af0cf4cb54922ad6d1f5de12a46e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.026798 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.026938 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d8ce12926e044ee95291a5e9168277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.027287 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f68a0c4bb341a9a90cc2be0f78daca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.027346 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47980e0c9ad487b8113f13458e175ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.027694 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391b412dbb9e420c90aa79473bbbe8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.028121 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc136dc84de44b49abcdf4100a86137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.028545 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7bf419440645279046b030e7554893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.028982 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710ad7e9c9664da9ab53f13455ba1256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.029410 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47fa53b050494194947585dbca53f335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.029537 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391b412dbb9e420c90aa79473bbbe8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.029819 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.029932 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20841d3424fb4c48b4de4ed7b2917abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.030395 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7bf419440645279046b030e7554893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.030722 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710ad7e9c9664da9ab53f13455ba1256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.031072 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47fa53b050494194947585dbca53f335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.031473 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c450ae4c434e4c6fa73d06c184b2bdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.031464 882 tablet_service.cc:1468] Processing CreateTablet for tablet f47980e0c9ad487b8113f13458e175ec (DEFAULT_TABLE table=default.Table_fec914ebb830498da551091c86a181cd [id=0fce93fe153745e6826dc4aec9b00186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.031956 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d8ce12926e044ee95291a5e9168277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.032024 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c450ae4c434e4c6fa73d06c184b2bdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.032568 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410e7c1ea44c42429a02bac3b3dd8b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.033068 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc136dc84de44b49abcdf4100a86137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.035179 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8faa58acd09649eab1a622e8c1ddb224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.036842 1165 ts_tablet_manager.cc:1918] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.037070 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_86e757b5d1f74c6a8d94cae99ec7a974" } new_table_name: "default.Table_d4ecf658368d43e2ba8c6e755e51b4e3" modify_external_catalogs: true
I20250625 13:52:13.037321 1165 log.cc:1199] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/2de9656958164dc4ae2fb0078b38f6c3
I20250625 13:52:13.037672 1165 ts_tablet_manager.cc:1939] T 2de9656958164dc4ae2fb0078b38f6c3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:13.037108 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0af0cf4cb54922ad6d1f5de12a46e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.038564 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3ed19313833f4518bd076c28a982830a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.039057 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ed19313833f4518bd076c28a982830a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.043349 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_45daef3051ac480facff13fca3be7f75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.048249 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 9df1f195befe4aed999b41f770547fde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.048344 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_18b980dca5124627abd9e0aa0476057d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.048462 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a1263945be894618a7f9946e7ca50efd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.048573 885 tablet_service.cc:1515] Processing DeleteTablet for tablet fd7a6587aa674a0eaaa107e2a2f673a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.048681 885 tablet_service.cc:1515] Processing DeleteTablet for tablet dfaba56442c0498da95e6fad126ab86d with delete_type TABLET_DATA_DELETED (replaced by tablet ee09a3942acc41acb4f7c0d5452bf268) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.049134 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7222541aef894753bea095d602e31b74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.055564 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7222541aef894753bea095d602e31b74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:13.056754 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45daef3051ac480facff13fca3be7f75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:13.057618 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18b980dca5124627abd9e0aa0476057d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.058372 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 909f4bb91e784b29acbbc9763e35ab03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.060828 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_eada3fe891ee4be3aeca6c124a5ad5a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.061301 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eada3fe891ee4be3aeca6c124a5ad5a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.072046 1171 tablet_bootstrap.cc:492] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:13.074527 1166 tablet_bootstrap.cc:492] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:13.074873 1166 ts_tablet_manager.cc:1397] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.195s user 0.014s sys 0.016s
I20250625 13:52:13.076498 1166 raft_consensus.cc:357] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.077121 1166 raft_consensus.cc:383] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.077301 1166 raft_consensus.cc:738] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.077769 1166 consensus_queue.cc:260] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.079944 1166 ts_tablet_manager.cc:1428] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250625 13:52:13.080704 1166 tablet_bootstrap.cc:492] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:13.083918 1166 tablet_bootstrap.cc:654] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.088439 2210 raft_consensus.cc:491] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.088977 2210 raft_consensus.cc:513] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:13.090602 851 leader_election.cc:343] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a0e81d4c904248eba858462bcf31c5d9
I20250625 13:52:13.091511 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0e81d4c904248eba858462bcf31c5d9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:13.091734 1047 raft_consensus.cc:2466] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:13.092229 849 leader_election.cc:304] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:13.092465 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c5d44351d1f8484294462de887c97b18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.092619 2271 raft_consensus.cc:2802] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:13.092695 2271 raft_consensus.cc:491] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:13.092767 2271 raft_consensus.cc:3058] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:13.093034 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5d44351d1f8484294462de887c97b18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.094357 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0a691ba3cecb4068b1955aa46a0ead68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.094938 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a691ba3cecb4068b1955aa46a0ead68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.095021 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_01f1a91fcc43403db4e9c2092f12a888" } new_table_name: "default.Table_a8bf7ce4386148c7a5d4880ccff0d3ca" modify_external_catalogs: true
I20250625 13:52:13.095819 2210 leader_election.cc:290] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:13.096190 1171 ts_tablet_manager.cc:1397] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.398s user 0.005s sys 0.008s
W20250625 13:52:13.096879 2162 log.cc:927] Time spent T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
W20250625 13:52:13.097091 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a44fab4e30564e9fbbc8d2b351437052 consensus state has no leader
I20250625 13:52:13.099509 1166 tablet_bootstrap.cc:492] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:13.100247 1166 ts_tablet_manager.cc:1397] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.006s
I20250625 13:52:13.103847 2210 raft_consensus.cc:491] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.103986 2210 raft_consensus.cc:513] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.104351 1166 raft_consensus.cc:357] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:13.105357 851 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 6dd84b8afa2548fe8ce5925f0bf806f0
I20250625 13:52:13.105247 1166 raft_consensus.cc:383] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.105613 1166 raft_consensus.cc:738] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.106122 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd84b8afa2548fe8ce5925f0bf806f0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:13.106321 1047 raft_consensus.cc:2466] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:13.106745 849 leader_election.cc:304] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:13.107010 2294 raft_consensus.cc:2802] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 13:52:13.107087 2294 raft_consensus.cc:491] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:13.107151 2294 raft_consensus.cc:3058] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:52:13.106739 1166 consensus_queue.cc:260] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.110670 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ef2cb3b80b6640c6a511b88a051fa7b4" } new_table_name: "default.Table_95a8e511ed19420c80b8dec20bce4adf" modify_external_catalogs: true
I20250625 13:52:13.109256 1166 ts_tablet_manager.cc:1428] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20250625 13:52:13.112624 1166 tablet_bootstrap.cc:492] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:13.122499 1166 tablet_bootstrap.cc:654] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.130673 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d070a8c36af544c6b3795695527477aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.131336 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d070a8c36af544c6b3795695527477aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.132577 2271 raft_consensus.cc:513] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.133589 2271 leader_election.cc:290] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:13.135533 774 raft_consensus.cc:2466] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:13.137116 1166 tablet_bootstrap.cc:492] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:13.137809 1166 ts_tablet_manager.cc:1397] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20250625 13:52:13.139261 2210 leader_election.cc:290] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:13.142904 1166 raft_consensus.cc:357] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.143831 1166 raft_consensus.cc:383] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.144106 1166 raft_consensus.cc:738] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.145207 1166 consensus_queue.cc:260] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.147498 1166 ts_tablet_manager.cc:1428] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250625 13:52:13.148633 1171 raft_consensus.cc:357] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.148818 1171 raft_consensus.cc:383] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.148880 1171 raft_consensus.cc:738] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.149076 1171 consensus_queue.cc:260] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.149467 1170 tablet_bootstrap.cc:492] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:13.149570 1171 ts_tablet_manager.cc:1428] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.053s user 0.001s sys 0.001s
I20250625 13:52:13.149683 1170 ts_tablet_manager.cc:1397] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.153s user 0.023s sys 0.014s
I20250625 13:52:13.149839 1171 tablet_bootstrap.cc:492] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:13.151100 1170 raft_consensus.cc:357] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.151413 1170 raft_consensus.cc:383] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.151508 1170 raft_consensus.cc:738] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.152223 1170 consensus_queue.cc:260] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.160725 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b14d00c93a964edd815122e3723f89e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.163883 1166 tablet_bootstrap.cc:492] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:13.164328 1170 ts_tablet_manager.cc:1428] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250625 13:52:13.177182 1170 tablet_bootstrap.cc:492] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:13.178103 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47fa53b050494194947585dbca53f335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.178601 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47980e0c9ad487b8113f13458e175ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.178941 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d8ce12926e044ee95291a5e9168277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.179271 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8faa58acd09649eab1a622e8c1ddb224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.179633 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc136dc84de44b49abcdf4100a86137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.179822 1171 tablet_bootstrap.cc:654] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.179965 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 391b412dbb9e420c90aa79473bbbe8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.180326 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710ad7e9c9664da9ab53f13455ba1256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.180657 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.181025 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0af0cf4cb54922ad6d1f5de12a46e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.182603 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410e7c1ea44c42429a02bac3b3dd8b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.182979 2204 raft_consensus.cc:491] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.183122 2204 raft_consensus.cc:513] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.183750 2204 leader_election.cc:290] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:13.183773 1170 tablet_bootstrap.cc:654] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.183724 851 leader_election.cc:304] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:13.183710 1166 tablet_bootstrap.cc:654] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.184262 2204 raft_consensus.cc:2802] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:13.184376 2204 raft_consensus.cc:695] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:13.184603 2204 consensus_queue.cc:237] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.186587 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "490c8902efeb46c99543747a8787c1c7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:13.187234 849 leader_election.cc:343] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.187706 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0e81d4c904248eba858462bcf31c5d9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:13.187852 1047 raft_consensus.cc:3058] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:13.188290 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_963851a3d1c4472eab0949b2c70d4279" } modify_external_catalogs: true
I20250625 13:52:13.188443 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_963851a3d1c4472eab0949b2c70d4279" } modify_external_catalogs: true
I20250625 13:52:13.191506 2300 raft_consensus.cc:491] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.191653 2300 raft_consensus.cc:513] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.192320 2300 leader_election.cc:290] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:13.193270 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_06d8369e7a09451eba488006b53da953" } modify_external_catalogs: true
I20250625 13:52:13.193423 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_06d8369e7a09451eba488006b53da953" } modify_external_catalogs: true
I20250625 13:52:13.193804 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 2de9656958164dc4ae2fb0078b38f6c3 (table default.Table_89e73d4233d14f7ab1805e3351155943 [id=fe513f20709d46d3832d046a1544c47f]) successfully deleted
I20250625 13:52:13.193894 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f2fba4b44d8e4e01994d0e99459c25f6" } modify_external_catalogs: true
I20250625 13:52:13.194036 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f2fba4b44d8e4e01994d0e99459c25f6" } modify_external_catalogs: true
I20250625 13:52:13.194356 1165 tablet_replica.cc:331] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:52:13.194435 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 466717307dd04d1cbdac9f03229f6a3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 466717307dd04d1cbdac9f03229f6a3b already in progress: creating tablet
I20250625 13:52:13.194532 1165 raft_consensus.cc:2241] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:13.194617 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 909f4bb91e784b29acbbc9763e35ab03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 909f4bb91e784b29acbbc9763e35ab03 already in progress: creating tablet
I20250625 13:52:13.194684 1165 raft_consensus.cc:2270] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:13.194782 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9df1f195befe4aed999b41f770547fde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9df1f195befe4aed999b41f770547fde already in progress: creating tablet
W20250625 13:52:13.194936 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a1263945be894618a7f9946e7ca50efd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1263945be894618a7f9946e7ca50efd already in progress: creating tablet
W20250625 13:52:13.195080 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet fd7a6587aa674a0eaaa107e2a2f673a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd7a6587aa674a0eaaa107e2a2f673a3 already in progress: creating tablet
W20250625 13:52:13.195158 985 leader_election.cc:343] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9c1deb52366c404cbdc08076d4bcb3d1
I20250625 13:52:13.195199 1165 ts_tablet_manager.cc:1905] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.202848 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c450ae4c434e4c6fa73d06c184b2bdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.203397 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7bf419440645279046b030e7554893. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.203786 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20841d3424fb4c48b4de4ed7b2917abe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:13.185356 851 leader_election.cc:343] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a0e81d4c904248eba858462bcf31c5d9
I20250625 13:52:13.213902 1166 tablet_bootstrap.cc:492] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:13.214169 1166 ts_tablet_manager.cc:1397] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.000s
I20250625 13:52:13.198113 1168 ts_tablet_manager.cc:1918] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.215663 1166 raft_consensus.cc:357] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.215920 1168 log.cc:1199] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/024f6cb363594420b0d5480fa79d156b
I20250625 13:52:13.215935 1166 raft_consensus.cc:383] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.216020 1166 raft_consensus.cc:738] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.216490 1166 consensus_queue.cc:260] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.217320 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47980e0c9ad487b8113f13458e175ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.218502 1170 tablet_bootstrap.cc:492] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:13.218737 1170 ts_tablet_manager.cc:1397] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.007s
I20250625 13:52:13.220610 1170 raft_consensus.cc:357] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.221021 1170 raft_consensus.cc:383] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.221168 1170 raft_consensus.cc:738] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.222151 1170 consensus_queue.cc:260] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.230350 1168 ts_tablet_manager.cc:1939] T 024f6cb363594420b0d5480fa79d156b P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:13.230698 851 consensus_peers.cc:487] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 067272869c5547648c90f83a34a036aa. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:13.230882 851 leader_election.cc:343] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 490c8902efeb46c99543747a8787c1c7
I20250625 13:52:13.230965 851 leader_election.cc:304] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:13.231911 983 leader_election.cc:343] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 9c1deb52366c404cbdc08076d4bcb3d1
I20250625 13:52:13.232023 983 leader_election.cc:304] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:13.232399 2300 raft_consensus.cc:2747] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:13.222824 1166 ts_tablet_manager.cc:1428] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250625 13:52:13.233462 1170 ts_tablet_manager.cc:1428] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20250625 13:52:13.233996 1170 tablet_bootstrap.cc:492] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:13.230708 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 819) took 4636 ms. Trace:
I20250625 13:52:13.236812 1170 tablet_bootstrap.cc:654] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.239634 2192 raft_consensus.cc:2747] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:13.244791 2266 tablet.cc:1722] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:13.244484 2192 consensus_queue.cc:1035] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20250625 13:52:13.236127 1168 rpcz_store.cc:276] 0625 13:52:08.594547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.780359 (+185812us) service_pool.cc:225] Handling call
0625 13:52:12.806307 (+4025948us) ts_tablet_manager.cc:1145] Deleting tablet 024f6cb363594420b0d5480fa79d156b
0625 13:52:12.806315 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.024077 (+217762us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.197987 (+173910us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.230693 (+ 32706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4025714}
I20250625 13:52:13.263659 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 024f6cb363594420b0d5480fa79d156b (table default.Table_81c1b426fda24349805e3e5cf9fdd561 [id=3eae0d10bcb247e0a318d6ac81e8c5da]) successfully deleted
I20250625 13:52:13.264091 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 835) took 4275 ms. Trace:
I20250625 13:52:13.264199 1168 rpcz_store.cc:276] 0625 13:52:08.988572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.988952 (+ 380us) service_pool.cc:225] Handling call
0625 13:52:13.263191 (+4274239us) ts_tablet_manager.cc:1145] Deleting tablet 9aadc67058554e87bb58d14b911333c4
0625 13:52:13.263201 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.264067 (+ 866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274028}
I20250625 13:52:13.265280 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 836) took 4273 ms. Trace:
I20250625 13:52:13.265753 1168 rpcz_store.cc:276] 0625 13:52:08.991601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:08.991656 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:13.265170 (+4273514us) ts_tablet_manager.cc:1145] Deleting tablet c9a82adc5aa045f49e092dababafe4f8
0625 13:52:13.265180 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.265263 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4273288}
I20250625 13:52:13.266229 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 838) took 4246 ms. Trace:
I20250625 13:52:13.266335 1168 rpcz_store.cc:276] 0625 13:52:09.019682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.019734 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:13.266129 (+4246395us) ts_tablet_manager.cc:1145] Deleting tablet e1b4da58289342d991cdf278005b96a0
0625 13:52:13.266139 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.266214 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4246151}
W20250625 13:52:13.266755 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9aadc67058554e87bb58d14b911333c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aadc67058554e87bb58d14b911333c4 already in progress: creating tablet
W20250625 13:52:13.267019 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c9a82adc5aa045f49e092dababafe4f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a82adc5aa045f49e092dababafe4f8 already in progress: creating tablet
W20250625 13:52:13.267529 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e1b4da58289342d991cdf278005b96a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1b4da58289342d991cdf278005b96a0 already in progress: creating tablet
I20250625 13:52:13.269950 1171 tablet_bootstrap.cc:492] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:13.270153 1171 ts_tablet_manager.cc:1397] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.120s user 0.000s sys 0.004s
I20250625 13:52:13.272552 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 837) took 4253 ms. Trace:
I20250625 13:52:13.272939 1168 rpcz_store.cc:276] 0625 13:52:09.019362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.020017 (+ 655us) service_pool.cc:225] Handling call
0625 13:52:13.267404 (+4247387us) ts_tablet_manager.cc:1145] Deleting tablet daa16c7ca9c7480dbd5fbd8114a449f2
0625 13:52:13.267412 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.272532 (+ 5120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4247223}
I20250625 13:52:13.274499 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 864) took 3737 ms. Trace:
I20250625 13:52:13.274775 1168 rpcz_store.cc:276] 0625 13:52:09.536864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.540914 (+ 4050us) service_pool.cc:225] Handling call
0625 13:52:13.274068 (+3733154us) ts_tablet_manager.cc:1145] Deleting tablet 8b0e49ca4dee42c59c0a2927a47aa320
0625 13:52:13.274115 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.274391 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3732831}
W20250625 13:52:13.275327 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet daa16c7ca9c7480dbd5fbd8114a449f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daa16c7ca9c7480dbd5fbd8114a449f2 already in progress: creating tablet
I20250625 13:52:13.275683 1168 tablet_replica.cc:331] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:13.279601 592 catalog_manager.cc:5582] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:13.279980 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
W20250625 13:52:13.280334 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c84b93a9079e45bab55353734ebe7cc2 consensus state has no leader
I20250625 13:52:13.281951 593 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:13.282590 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e460144bc7964ddba939c069278e887d" } modify_external_catalogs: true
I20250625 13:52:13.282722 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e460144bc7964ddba939c069278e887d" } modify_external_catalogs: true
I20250625 13:52:13.283553 588 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (table default.Table_6d7df5c325b44ea1899f5faa70e7c37d [id=3977ee3968d246b7a8aed1fafd1822b7]). Expected version 2 got 0
I20250625 13:52:13.284166 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_73b9f78abc404d3c98a2a57d96a30f84" } modify_external_catalogs: true
I20250625 13:52:13.284299 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_73b9f78abc404d3c98a2a57d96a30f84" } modify_external_catalogs: true
I20250625 13:52:13.292640 745 tablet_service.cc:1468] Processing CreateTablet for tablet 8e4df287c95946e2beb94524073e5ee2 (DEFAULT_TABLE table=default.Table_cf58706a138644a49767bbb2201a120e [id=f5a2e1e208ef4bd593ed2879f9d55129]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.292896 753 tablet_service.cc:1468] Processing CreateTablet for tablet 1400e8320d7f470a844ac3eec7e60d8c (DEFAULT_TABLE table=default.Table_3d5b64d4b9ce4cd3a8b4625159a26e70 [id=b2763ba794084b16adc05ddbc1cc3a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.293721 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4211d1a18c624dde9c565da4070397e4 (DEFAULT_TABLE table=default.Table_021ec65b75b74e9db71be9078b4a1067 [id=67676561a259430299d98a2c6f9fd5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.294909 757 tablet_service.cc:1468] Processing CreateTablet for tablet c37fab09d2014b0b97dc17216dbc0bf5 (DEFAULT_TABLE table=default.Table_0af46ccac2054224a7c0186bbc940d21 [id=12f3de56154643299cd8b3899711377e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.295831 756 tablet_service.cc:1468] Processing CreateTablet for tablet d37ef6d93c8e44e985882723cce2d990 (DEFAULT_TABLE table=default.Table_a4a8e4337fea4493bd07b556314e49db [id=4ce18d456651499281dca929c9c8279b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.299523 751 tablet_service.cc:1468] Processing CreateTablet for tablet 18014e8cb05a44e694e9c36d7a2aecbc (DEFAULT_TABLE table=default.Table_f5c726e169ca41b6a7092eef2e923eb5 [id=70585a5b7299452fa9015570c13b5a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.308887 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:13.315840 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 8e4df287c95946e2beb94524073e5ee2 (DEFAULT_TABLE table=default.Table_cf58706a138644a49767bbb2201a120e [id=f5a2e1e208ef4bd593ed2879f9d55129]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.316838 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 1400e8320d7f470a844ac3eec7e60d8c (DEFAULT_TABLE table=default.Table_3d5b64d4b9ce4cd3a8b4625159a26e70 [id=b2763ba794084b16adc05ddbc1cc3a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.317515 1170 tablet_bootstrap.cc:492] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:13.317641 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 4211d1a18c624dde9c565da4070397e4 (DEFAULT_TABLE table=default.Table_021ec65b75b74e9db71be9078b4a1067 [id=67676561a259430299d98a2c6f9fd5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.318130 1170 ts_tablet_manager.cc:1397] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.084s user 0.017s sys 0.020s
I20250625 13:52:13.318414 1022 tablet_service.cc:1468] Processing CreateTablet for tablet c37fab09d2014b0b97dc17216dbc0bf5 (DEFAULT_TABLE table=default.Table_0af46ccac2054224a7c0186bbc940d21 [id=12f3de56154643299cd8b3899711377e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.319988 1023 tablet_service.cc:1468] Processing CreateTablet for tablet d37ef6d93c8e44e985882723cce2d990 (DEFAULT_TABLE table=default.Table_a4a8e4337fea4493bd07b556314e49db [id=4ce18d456651499281dca929c9c8279b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.320394 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 18014e8cb05a44e694e9c36d7a2aecbc (DEFAULT_TABLE table=default.Table_f5c726e169ca41b6a7092eef2e923eb5 [id=70585a5b7299452fa9015570c13b5a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.320870 1014 tablet_service.cc:1468] Processing CreateTablet for tablet ce47e254a7a54d53ae87fc6de392eb2d (DEFAULT_TABLE table=default.Table_d41a791307e142c3b46cd3fda9482764 [id=ff66996ea13846218b3e8bfa58993241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.322031 1170 raft_consensus.cc:357] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.322711 1170 raft_consensus.cc:383] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.323081 1170 raft_consensus.cc:738] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.324332 2254 ts_tablet_manager.cc:1918] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.324432 2254 log.cc:1199] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/3b0037d09796455cbaaadfaa9c010673
I20250625 13:52:13.324347 2303 tablet.cc:1722] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:13.324712 2254 ts_tablet_manager.cc:1939] T 3b0037d09796455cbaaadfaa9c010673 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:13.325122 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18014e8cb05a44e694e9c36d7a2aecbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.324258 1170 consensus_queue.cc:260] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.325631 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37ef6d93c8e44e985882723cce2d990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.326061 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce47e254a7a54d53ae87fc6de392eb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.329001 1170 ts_tablet_manager.cc:1428] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250625 13:52:13.330330 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c37fab09d2014b0b97dc17216dbc0bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.330847 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4211d1a18c624dde9c565da4070397e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.330770 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 40983488884a414e82871141112bd8b2 (DEFAULT_TABLE table=default.Table_ac4439707ad3409595f54dce3c9b810f [id=4b7fe0ca3a3044a0b82fd38b4bfa191a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.330893 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb3e53b6e7e427cac7adefb2cbde4c0 (DEFAULT_TABLE table=default.Table_0837b1ebf17d4ba4a4b0479fe815f9b6 [id=66a91ec72e5c4da699a155ad56335498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.331876 1170 tablet_bootstrap.cc:492] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:13.333693 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 95dfea44fd4a4591b9363670b2ababe9 (DEFAULT_TABLE table=default.Table_77d42655ef0349128a8518e3892d4532 [id=29d11ad122834f9fa197f1ff2f27b3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.335443 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1400e8320d7f470a844ac3eec7e60d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.335868 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4df287c95946e2beb94524073e5ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.340965 1047 raft_consensus.cc:2466] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:13.349658 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8e4df287c95946e2beb94524073e5ee2 (DEFAULT_TABLE table=default.Table_cf58706a138644a49767bbb2201a120e [id=f5a2e1e208ef4bd593ed2879f9d55129]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.350103 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4df287c95946e2beb94524073e5ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.356858 744 tablet_service.cc:1468] Processing CreateTablet for tablet ce47e254a7a54d53ae87fc6de392eb2d (DEFAULT_TABLE table=default.Table_d41a791307e142c3b46cd3fda9482764 [id=ff66996ea13846218b3e8bfa58993241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.357262 740 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb3e53b6e7e427cac7adefb2cbde4c0 (DEFAULT_TABLE table=default.Table_0837b1ebf17d4ba4a4b0479fe815f9b6 [id=66a91ec72e5c4da699a155ad56335498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.357560 739 tablet_service.cc:1468] Processing CreateTablet for tablet 40983488884a414e82871141112bd8b2 (DEFAULT_TABLE table=default.Table_ac4439707ad3409595f54dce3c9b810f [id=4b7fe0ca3a3044a0b82fd38b4bfa191a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.358539 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1400e8320d7f470a844ac3eec7e60d8c (DEFAULT_TABLE table=default.Table_3d5b64d4b9ce4cd3a8b4625159a26e70 [id=b2763ba794084b16adc05ddbc1cc3a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.358906 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1400e8320d7f470a844ac3eec7e60d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.363610 2300 raft_consensus.cc:491] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.363739 2300 raft_consensus.cc:513] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.364291 2300 leader_election.cc:290] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:13.365866 985 leader_election.cc:343] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b14d00c93a964edd815122e3723f89e7
I20250625 13:52:13.367858 1165 ts_tablet_manager.cc:1918] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.368006 1165 log.cc:1199] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/dfaba56442c0498da95e6fad126ab86d
I20250625 13:52:13.368300 1165 ts_tablet_manager.cc:1939] T dfaba56442c0498da95e6fad126ab86d P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:13.370033 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4df287c95946e2beb94524073e5ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.370275 749 tablet_service.cc:1468] Processing CreateTablet for tablet 95dfea44fd4a4591b9363670b2ababe9 (DEFAULT_TABLE table=default.Table_77d42655ef0349128a8518e3892d4532 [id=29d11ad122834f9fa197f1ff2f27b3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.370622 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95dfea44fd4a4591b9363670b2ababe9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:13.375619 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 170ffbff76af447a83d663fa8688b7a2 consensus state has no leader
I20250625 13:52:13.375648 2322 tablet.cc:1722] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 13:52:13.375973 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1e94167cff8f4ee8ad52fa7f9e012a35" } new_table_name: "default.Table_c9f8713b891a4605a583946a0fa51b1e" modify_external_catalogs: true
I20250625 13:52:13.382980 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e899e0383341441d9eeca197fd80a8be" } new_table_name: "default.Table_d6b292db8ea248f792aaab082b9adfa4" modify_external_catalogs: true
I20250625 13:52:13.383749 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_154bff2914d341e5b12048cfbb81b179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.384220 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_154bff2914d341e5b12048cfbb81b179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.386570 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 71170aaf26ed4f1b8c458d90db6e2765
I20250625 13:52:13.387239 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2a88884ec2db4ae8bc4e94280d4c5022" } new_table_name: "default.Table_995a32a383304336af2581d84d9db1e0" modify_external_catalogs: true
I20250625 13:52:13.389796 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 40f7656d6e53489ab982ba96bc04771b
I20250625 13:52:13.390300 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_932a498cb59e4c8bae0789feef7c91e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.390709 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_932a498cb59e4c8bae0789feef7c91e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.391541 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 40f7656d6e53489ab982ba96bc04771b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:13.391633 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d36d7fc8ec7d4bbeb6f5445c8c1b210d consensus state has no leader
I20250625 13:52:13.391919 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_28183b013d3c41b4a66d5037fd020e5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.392343 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28183b013d3c41b4a66d5037fd020e5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.393886 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a2ebcb65c49746f3ba9aa905e657c107
I20250625 13:52:13.394425 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ec05413cb661420684670fce5e9b6107"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.394676 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ebcb65c49746f3ba9aa905e657c107 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:13.394819 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec05413cb661420684670fce5e9b6107 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.402470 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (table default.Table_6d7df5c325b44ea1899f5faa70e7c37d [id=3977ee3968d246b7a8aed1fafd1822b7]). Expected version 2 got 0
I20250625 13:52:13.403913 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 71170aaf26ed4f1b8c458d90db6e2765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.404848 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3713b85743d04649b27517412b3d9ea0
I20250625 13:52:13.405699 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6bd08e23382b4c0880724d8266377783"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.406675 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bd08e23382b4c0880724d8266377783 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.408362 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 3713b85743d04649b27517412b3d9ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.409937 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_232257f7ad2d4eb689c206dfde0510fd [id=71729b0c8f7046bb8e80bb96f9f2ba71] alter complete (version 3)
I20250625 13:52:13.410619 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c72bc73875ff422da063a9cce3941db4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.411237 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c72bc73875ff422da063a9cce3941db4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.418831 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 569f6cba69394f2fa005686056f19929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.418946 595 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:13.419030 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.419351 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 569f6cba69394f2fa005686056f19929
I20250625 13:52:13.419811 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 171ce24a435e40e48026039941ca7883 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:13.420157 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 569f6cba69394f2fa005686056f19929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.420312 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e85e50c4b7e144caa0afdfc419814cae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.420645 1170 tablet_bootstrap.cc:654] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:13.420799 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e85e50c4b7e144caa0afdfc419814cae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.423384 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.427726 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 40f7656d6e53489ab982ba96bc04771b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.429075 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 71170aaf26ed4f1b8c458d90db6e2765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.434181 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ebcb65c49746f3ba9aa905e657c107 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.434405 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3713b85743d04649b27517412b3d9ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.434605 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 569f6cba69394f2fa005686056f19929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.435755 1166 tablet_bootstrap.cc:492] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:13.437356 1166 tablet_bootstrap.cc:654] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.439407 889 tablet_service.cc:1468] Processing CreateTablet for tablet 4211d1a18c624dde9c565da4070397e4 (DEFAULT_TABLE table=default.Table_021ec65b75b74e9db71be9078b4a1067 [id=67676561a259430299d98a2c6f9fd5bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:13.443667 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78c79a2f6544493ba3f3ec20ea67fa95 consensus state has no leader
W20250625 13:52:13.446200 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c930ac853a304fbab4be6fb37df5f79a consensus state has no leader
I20250625 13:52:13.447793 2294 raft_consensus.cc:513] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.448563 2294 leader_election.cc:290] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:13.449538 1168 raft_consensus.cc:2241] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.449862 1168 raft_consensus.cc:2270] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.450448 1168 ts_tablet_manager.cc:1905] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.456815 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:13.456782 592 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_934cbfc5dfe44861a26a970575a6facb [id=9e881be4b5c54f5191713d2fb732d716] alter complete (version 2)
I20250625 13:52:13.462289 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4211d1a18c624dde9c565da4070397e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.466830 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37ef6d93c8e44e985882723cce2d990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.467857 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1053) took 1396 ms. Trace:
I20250625 13:52:13.467945 2254 rpcz_store.cc:276] 0625 13:52:12.071803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.072667 (+ 864us) service_pool.cc:225] Handling call
0625 13:52:12.821172 (+748505us) ts_tablet_manager.cc:1145] Deleting tablet 3b0037d09796455cbaaadfaa9c010673
0625 13:52:12.821180 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.208252 (+387072us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.324242 (+115990us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.467835 (+143593us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":748363}
I20250625 13:52:13.468412 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1047) took 1397 ms. Trace:
I20250625 13:52:13.468479 2254 rpcz_store.cc:276] 0625 13:52:12.071261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.072757 (+ 1496us) service_pool.cc:225] Handling call
0625 13:52:13.468295 (+1395538us) ts_tablet_manager.cc:1145] Deleting tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604
0625 13:52:13.468305 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.468396 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395395}
I20250625 13:52:13.468710 2254 tablet_replica.cc:331] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:13.468886 2254 raft_consensus.cc:2241] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.469029 2254 raft_consensus.cc:2270] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.469501 2254 ts_tablet_manager.cc:1905] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.471664 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1400e8320d7f470a844ac3eec7e60d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.475543 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c37fab09d2014b0b97dc17216dbc0bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.479508 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4211d1a18c624dde9c565da4070397e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.483505 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18014e8cb05a44e694e9c36d7a2aecbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.483695 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3e53b6e7e427cac7adefb2cbde4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.486124 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40983488884a414e82871141112bd8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.486974 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95dfea44fd4a4591b9363670b2ababe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.487221 1171 raft_consensus.cc:357] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.487427 1171 raft_consensus.cc:383] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.487484 1171 raft_consensus.cc:738] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.487709 1171 consensus_queue.cc:260] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.488197 1171 ts_tablet_manager.cc:1428] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.218s user 0.002s sys 0.000s
I20250625 13:52:13.495595 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce47e254a7a54d53ae87fc6de392eb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.499171 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40983488884a414e82871141112bd8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.500028 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3e53b6e7e427cac7adefb2cbde4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.500494 1165 tablet_replica.cc:331] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:13.500675 1165 raft_consensus.cc:2241] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.500818 1165 raft_consensus.cc:2270] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.501339 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet dfaba56442c0498da95e6fad126ab86d (table default.Table_f4de7823c10e40e1bdd42b0e0559f7bf [id=7270b5aa9a0148178683a229db5b4b7a]) successfully deleted
W20250625 13:52:13.501595 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 40f7656d6e53489ab982ba96bc04771b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40f7656d6e53489ab982ba96bc04771b already in progress: creating tablet
W20250625 13:52:13.501746 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a2ebcb65c49746f3ba9aa905e657c107 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ebcb65c49746f3ba9aa905e657c107 already in progress: creating tablet
I20250625 13:52:13.501865 1165 ts_tablet_manager.cc:1905] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:13.501876 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 71170aaf26ed4f1b8c458d90db6e2765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71170aaf26ed4f1b8c458d90db6e2765 already in progress: creating tablet
W20250625 13:52:13.502031 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3713b85743d04649b27517412b3d9ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3713b85743d04649b27517412b3d9ea0 already in progress: creating tablet
I20250625 13:52:13.505342 2254 ts_tablet_manager.cc:1918] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.505445 2254 log.cc:1199] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/067272869c5547648c90f83a34a036aa
I20250625 13:52:13.505761 2254 ts_tablet_manager.cc:1939] T 067272869c5547648c90f83a34a036aa P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:13.506045 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1054) took 1434 ms. Trace:
I20250625 13:52:13.506110 2254 rpcz_store.cc:276] 0625 13:52:12.071882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.072850 (+ 968us) service_pool.cc:225] Handling call
0625 13:52:13.468671 (+1395821us) ts_tablet_manager.cc:1145] Deleting tablet 067272869c5547648c90f83a34a036aa
0625 13:52:13.468678 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.473812 (+ 5134us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.505269 (+ 31457us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.506033 (+ 764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395706}
I20250625 13:52:13.506379 2254 tablet_replica.cc:331] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:13.506531 2254 raft_consensus.cc:2241] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.506664 2254 raft_consensus.cc:2270] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.507128 2254 ts_tablet_manager.cc:1905] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.507805 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d6b292db8ea248f792aaab082b9adfa4" } new_table_name: "default.Table_efc036d6fcdd445c9420a616e7ee8469" modify_external_catalogs: true
I20250625 13:52:13.510046 879 tablet_service.cc:1468] Processing CreateTablet for tablet 18014e8cb05a44e694e9c36d7a2aecbc (DEFAULT_TABLE table=default.Table_f5c726e169ca41b6a7092eef2e923eb5 [id=70585a5b7299452fa9015570c13b5a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.511593 890 tablet_service.cc:1468] Processing CreateTablet for tablet ce47e254a7a54d53ae87fc6de392eb2d (DEFAULT_TABLE table=default.Table_d41a791307e142c3b46cd3fda9482764 [id=ff66996ea13846218b3e8bfa58993241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.511700 1171 tablet_bootstrap.cc:492] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:13.511924 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce47e254a7a54d53ae87fc6de392eb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.512408 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18014e8cb05a44e694e9c36d7a2aecbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.513147 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_39d5b8799785461d861d7d1ef1fba8e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.513620 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39d5b8799785461d861d7d1ef1fba8e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.517419 1168 ts_tablet_manager.cc:1918] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.517607 1168 log.cc:1199] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/6dd84b8afa2548fe8ce5925f0bf806f0
I20250625 13:52:13.518262 891 tablet_service.cc:1468] Processing CreateTablet for tablet c37fab09d2014b0b97dc17216dbc0bf5 (DEFAULT_TABLE table=default.Table_0af46ccac2054224a7c0186bbc940d21 [id=12f3de56154643299cd8b3899711377e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.518635 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c37fab09d2014b0b97dc17216dbc0bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.527688 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 3b0037d09796455cbaaadfaa9c010673 (table default.Table_6b3565f151aa48d4b4e6dbc686f0c8ae [id=6f1fa6d1db4e4abaae4d06299152224f]) successfully deleted
W20250625 13:52:13.527931 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 already in progress: creating tablet
I20250625 13:52:13.528690 878 tablet_service.cc:1468] Processing CreateTablet for tablet d37ef6d93c8e44e985882723cce2d990 (DEFAULT_TABLE table=default.Table_a4a8e4337fea4493bd07b556314e49db [id=4ce18d456651499281dca929c9c8279b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.529103 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d37ef6d93c8e44e985882723cce2d990. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.533006 894 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb3e53b6e7e427cac7adefb2cbde4c0 (DEFAULT_TABLE table=default.Table_0837b1ebf17d4ba4a4b0479fe815f9b6 [id=66a91ec72e5c4da699a155ad56335498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.533399 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3e53b6e7e427cac7adefb2cbde4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.533780 882 tablet_service.cc:1468] Processing CreateTablet for tablet 40983488884a414e82871141112bd8b2 (DEFAULT_TABLE table=default.Table_ac4439707ad3409595f54dce3c9b810f [id=4b7fe0ca3a3044a0b82fd38b4bfa191a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.534125 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40983488884a414e82871141112bd8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.536222 1168 ts_tablet_manager.cc:1939] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:13.536762 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 865) took 3988 ms. Trace:
W20250625 13:52:13.540716 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78c79a2f6544493ba3f3ec20ea67fa95 consensus state has no leader
I20250625 13:52:13.543570 1165 ts_tablet_manager.cc:1918] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.543779 1165 log.cc:1199] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/569f6cba69394f2fa005686056f19929
I20250625 13:52:13.544332 1165 ts_tablet_manager.cc:1939] T 569f6cba69394f2fa005686056f19929 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:13.546113 1165 tablet_replica.cc:331] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:13.546471 1165 raft_consensus.cc:2241] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.547789 1165 raft_consensus.cc:2270] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.550966 1166 tablet_bootstrap.cc:492] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:13.551147 1166 ts_tablet_manager.cc:1397] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.115s user 0.006s sys 0.011s
I20250625 13:52:13.551338 2254 ts_tablet_manager.cc:1918] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:13.552620 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 92984.9us
I20250625 13:52:13.560246 592 master_service.cc:948] ReplaceTablet: received request to replace tablet eabd67b9ceb74bd1a5f4f115c7db0e87 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:13.565284 1165 ts_tablet_manager.cc:1905] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.567032 1166 raft_consensus.cc:357] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.567229 1166 raft_consensus.cc:383] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.567338 1166 raft_consensus.cc:738] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.567562 1166 consensus_queue.cc:260] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.568095 1166 ts_tablet_manager.cc:1428] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 13:52:13.570676 2313 raft_consensus.cc:491] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.570801 2313 raft_consensus.cc:513] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.571738 2313 leader_election.cc:290] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:13.534421 893 tablet_service.cc:1468] Processing CreateTablet for tablet 95dfea44fd4a4591b9363670b2ababe9 (DEFAULT_TABLE table=default.Table_77d42655ef0349128a8518e3892d4532 [id=29d11ad122834f9fa197f1ff2f27b3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.574390 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95dfea44fd4a4591b9363670b2ababe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.548537 886 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ebcb65c49746f3ba9aa905e657c107 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.554527 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 71170aaf26ed4f1b8c458d90db6e2765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.569674 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 40f7656d6e53489ab982ba96bc04771b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.571409 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 569f6cba69394f2fa005686056f19929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.545002 1171 tablet_bootstrap.cc:654] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.576009 2204 raft_consensus.cc:491] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.576176 2204 raft_consensus.cc:513] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:13.576581 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d6710249ce844de2a9e8b63ae65de9de (table default.Table_ce4451ff102a4c22ba9539daffd65aa6 [id=4148344340d545ae8f071f8292500291], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d6710249ce844de2a9e8b63ae65de9de consensus state has no leader
I20250625 13:52:13.576576 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_06ce6be66e684fce8aa585310d55aba2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.577415 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ce6be66e684fce8aa585310d55aba2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.578589 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2e590dac73914afca2846962795030aa" } new_table_name: "default.Table_5d32664d42bc479397894fa7c57907f6" modify_external_catalogs: true
I20250625 13:52:13.578964 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 3713b85743d04649b27517412b3d9ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.582456 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:13.582633 2204 leader_election.cc:290] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:13.582715 592 catalog_manager.cc:5582] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:13.583029 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (table default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916]). Expected version 5 got 0
I20250625 13:52:13.585462 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c605d5b64c164c1594b0a40cfa2c4cf6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:13.585913 711 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.590718 739 tablet_service.cc:1515] Processing DeleteTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:13.592849 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d64d8f832da048e68e688a037b593719 consensus state has no leader
I20250625 13:52:13.593113 739 tablet_service.cc:1468] Processing CreateTablet for tablet bc8270f9ebd04fc39a320b87a0095a74 (DEFAULT_TABLE table=default.Table_eada3fe891ee4be3aeca6c124a5ad5a8 [id=68379b5865a44daaaaf58447895e3d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.593505 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8270f9ebd04fc39a320b87a0095a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.573833 1168 rpcz_store.cc:276] 0625 13:52:09.548415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.548525 (+ 110us) service_pool.cc:225] Handling call
0625 13:52:13.275542 (+3727017us) ts_tablet_manager.cc:1145] Deleting tablet 6dd84b8afa2548fe8ce5925f0bf806f0
0625 13:52:13.275551 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.449770 (+174219us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x6160000b5654. stack: 00007f37024dc336 00007f37024dc0f0 00007f37014d23ad 0000000000971004 0000000000970fcd 000000000097099a 00007f370df3c000 00007f370f24e093 00007f3711d2b137 00007f3711d5d1c4 00007f3711d3ec20 00007f3711d3e9ad 0000000000a7573c 00007f37025219df 00007f3702524cab 00007f3702524b28
0625 13:52:13.453803 (+ 4033us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.517318 (+ 63515us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.536728 (+ 19410us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":398216448,"tablet-delete.queue_time_us":3726751}
I20250625 13:52:13.594823 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 871) took 4001 ms. Trace:
I20250625 13:52:13.594902 1168 rpcz_store.cc:276] 0625 13:52:09.593664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.595499 (+ 1835us) service_pool.cc:225] Handling call
0625 13:52:13.594693 (+3999194us) ts_tablet_manager.cc:1145] Deleting tablet 29a542de77674e33b70647ffd9475fbf
0625 13:52:13.594704 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.594802 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3999028}
I20250625 13:52:13.595227 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 872) took 3984 ms. Trace:
I20250625 13:52:13.595288 1168 rpcz_store.cc:276] 0625 13:52:09.610960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.613721 (+ 2761us) service_pool.cc:225] Handling call
0625 13:52:13.595133 (+3981412us) ts_tablet_manager.cc:1145] Deleting tablet 39da3ebbe55f4c4fab4f8af444657b9b
0625 13:52:13.595142 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.595212 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3981232}
I20250625 13:52:13.595564 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 873) took 3959 ms. Trace:
I20250625 13:52:13.595625 1168 rpcz_store.cc:276] 0625 13:52:09.635690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.635748 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:13.595479 (+3959731us) ts_tablet_manager.cc:1145] Deleting tablet 3e60792b6796458c857415e258ece1c9
0625 13:52:13.595486 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.595553 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3959548}
I20250625 13:52:13.595835 1168 tablet_replica.cc:331] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:13.597345 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d1c8466871174daea32ea7d8bac86daa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.597952 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c72bc73875ff422da063a9cce3941db4" } modify_external_catalogs: true
I20250625 13:52:13.598102 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c72bc73875ff422da063a9cce3941db4" } modify_external_catalogs: true
I20250625 13:52:13.598136 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ef5aa5771b2b412ca7ebce55f40b6eed" } modify_external_catalogs: true
I20250625 13:52:13.598219 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ef5aa5771b2b412ca7ebce55f40b6eed" } modify_external_catalogs: true
W20250625 13:52:13.598356 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1c8466871174daea32ea7d8bac86daa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.599159 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_94aa82f5ff874e7b826fcb2abc61eeb7" } new_table_name: "default.Table_fac1b7cf47334e72b5815c78f0e35ba6" modify_external_catalogs: true
I20250625 13:52:13.601859 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_932a498cb59e4c8bae0789feef7c91e5" } modify_external_catalogs: true
I20250625 13:52:13.602000 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_932a498cb59e4c8bae0789feef7c91e5" } modify_external_catalogs: true
I20250625 13:52:13.602752 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 6dd84b8afa2548fe8ce5925f0bf806f0 (table default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]) successfully deleted
I20250625 13:52:13.603127 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.604480 756 tablet_service.cc:1468] Processing CreateTablet for tablet bf3168fb0cec45b5a45cd73eceee7414 (DEFAULT_TABLE table=default.Table_18b980dca5124627abd9e0aa0476057d [id=1174afee7936469e80fcd17a3d5afeed]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:13.602970 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 29a542de77674e33b70647ffd9475fbf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29a542de77674e33b70647ffd9475fbf already in progress: creating tablet
W20250625 13:52:13.605204 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 39da3ebbe55f4c4fab4f8af444657b9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39da3ebbe55f4c4fab4f8af444657b9b already in progress: creating tablet
W20250625 13:52:13.605444 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3e60792b6796458c857415e258ece1c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e60792b6796458c857415e258ece1c9 already in progress: creating tablet
I20250625 13:52:13.605854 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 20841d3424fb4c48b4de4ed7b2917abe
I20250625 13:52:13.606484 896 tablet_service.cc:1515] Processing DeleteTablet for tablet a29708fb688a4a65b84ae2564a040b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 2c8cf5d6cd624a1797486c69bd2098fd) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.606942 878 tablet_service.cc:1468] Processing CreateTablet for tablet bc8270f9ebd04fc39a320b87a0095a74 (DEFAULT_TABLE table=default.Table_eada3fe891ee4be3aeca6c124a5ad5a8 [id=68379b5865a44daaaaf58447895e3d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.607424 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8270f9ebd04fc39a320b87a0095a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.608574 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_61c4552a00dd4dfb882f0e86f3b8776a" } modify_external_catalogs: true
I20250625 13:52:13.608739 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_61c4552a00dd4dfb882f0e86f3b8776a" } modify_external_catalogs: true
I20250625 13:52:13.609947 592 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 63c438a383c542beb816048ec0e6d856
I20250625 13:52:13.613884 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c9ded52b36e04283b6145cfc6c750581
I20250625 13:52:13.614495 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7779da6edee64fad8cbd0131510bebd0 (DEFAULT_TABLE table=default.Table_167a0a50523949028824975d052ed256 [id=040cb0194b024efc8d97457be683b978]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.614964 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7779da6edee64fad8cbd0131510bebd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.554350 1170 tablet_bootstrap.cc:492] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:13.572276 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c605d5b64c164c1594b0a40cfa2c4cf6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:13.616806 2254 log.cc:1199] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/6dd84b8afa2548fe8ce5925f0bf806f0
I20250625 13:52:13.588233 1018 tablet_service.cc:1468] Processing CreateTablet for tablet bc8270f9ebd04fc39a320b87a0095a74 (DEFAULT_TABLE table=default.Table_eada3fe891ee4be3aeca6c124a5ad5a8 [id=68379b5865a44daaaaf58447895e3d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.617148 2254 ts_tablet_manager.cc:1939] T 6dd84b8afa2548fe8ce5925f0bf806f0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:13.617303 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8270f9ebd04fc39a320b87a0095a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.617187 896 tablet_service.cc:1468] Processing CreateTablet for tablet bf3168fb0cec45b5a45cd73eceee7414 (DEFAULT_TABLE table=default.Table_18b980dca5124627abd9e0aa0476057d [id=1174afee7936469e80fcd17a3d5afeed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.617600 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf3168fb0cec45b5a45cd73eceee7414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.618821 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1052) took 1547 ms. Trace:
I20250625 13:52:13.618911 2254 rpcz_store.cc:276] 0625 13:52:12.071692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.072929 (+ 1237us) service_pool.cc:225] Handling call
0625 13:52:13.506343 (+1433414us) ts_tablet_manager.cc:1145] Deleting tablet 6dd84b8afa2548fe8ce5925f0bf806f0
0625 13:52:13.506350 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.542061 (+ 35711us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.551233 (+ 9172us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.618807 (+ 67574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433312}
I20250625 13:52:13.603214 1030 tablet_service.cc:1468] Processing CreateTablet for tablet bf3168fb0cec45b5a45cd73eceee7414 (DEFAULT_TABLE table=default.Table_18b980dca5124627abd9e0aa0476057d [id=1174afee7936469e80fcd17a3d5afeed]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:13.619954 711 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.608990 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 20841d3424fb4c48b4de4ed7b2917abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.620085 711 leader_election.cc:304] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:13.609683 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 7779da6edee64fad8cbd0131510bebd0 (DEFAULT_TABLE table=default.Table_167a0a50523949028824975d052ed256 [id=040cb0194b024efc8d97457be683b978]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.620339 2322 tablet.cc:1722] T dad4ea8cf44944b08f53428c95809daf P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:13.620465 2313 raft_consensus.cc:2747] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:13.614560 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f827dd12cc3049ba843bb47cb9e412d9 (DEFAULT_TABLE table=default.Table_45daef3051ac480facff13fca3be7f75 [id=05bdba00acc94dfdaff1ecce7e0b3111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.623730 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 20841d3424fb4c48b4de4ed7b2917abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.625837 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b2017c57e5c7416494c31f6dcefdc43d" } modify_external_catalogs: true
I20250625 13:52:13.626113 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b2017c57e5c7416494c31f6dcefdc43d" } modify_external_catalogs: true
I20250625 13:52:13.627488 886 tablet_service.cc:1468] Processing CreateTablet for tablet f827dd12cc3049ba843bb47cb9e412d9 (DEFAULT_TABLE table=default.Table_45daef3051ac480facff13fca3be7f75 [id=05bdba00acc94dfdaff1ecce7e0b3111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.627939 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f827dd12cc3049ba843bb47cb9e412d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.628351 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9f24ae7c08bd42959ab48f7007ec98d3
I20250625 13:52:13.632256 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 20841d3424fb4c48b4de4ed7b2917abe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.632756 753 tablet_service.cc:1468] Processing CreateTablet for tablet 1d0f8626d2ac470891919ef051d2bd23 (DEFAULT_TABLE table=default.Table_a1144201285a4e62b6eedd9332730eb6 [id=74355b566c5f4e1aaf0883375522a419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.633231 1166 tablet_bootstrap.cc:492] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:13.633668 758 tablet_service.cc:1468] Processing CreateTablet for tablet f827dd12cc3049ba843bb47cb9e412d9 (DEFAULT_TABLE table=default.Table_45daef3051ac480facff13fca3be7f75 [id=05bdba00acc94dfdaff1ecce7e0b3111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.633989 751 tablet_service.cc:1468] Processing CreateTablet for tablet bf2dafb49e9d4003962ebddd16a50f76 (DEFAULT_TABLE table=default.Table_7222541aef894753bea095d602e31b74 [id=df2d817df2fc44e68b918280db6ebcdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.635172 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 6dd84b8afa2548fe8ce5925f0bf806f0 (table default.Table_bbd45439068f4c9188192e1b98a84784 [id=951cff6d2bf34c82b99049c683187a53]) successfully deleted
I20250625 13:52:13.636773 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f24ae7c08bd42959ab48f7007ec98d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.639415 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6d95a376304e464ab75e2e1e00e7a924
I20250625 13:52:13.635583 757 tablet_service.cc:1468] Processing CreateTablet for tablet 7779da6edee64fad8cbd0131510bebd0 (DEFAULT_TABLE table=default.Table_167a0a50523949028824975d052ed256 [id=040cb0194b024efc8d97457be683b978]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.640410 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d95a376304e464ab75e2e1e00e7a924 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.648909 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d95a376304e464ab75e2e1e00e7a924 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.649991 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f24ae7c08bd42959ab48f7007ec98d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.654760 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_41e5ba0098654109a19569d17b1ab1b9" } new_table_name: "default.Table_7f7b859cf51248448fca7a26ae20f8cd" modify_external_catalogs: true
I20250625 13:52:13.653126 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_be523f2a377f4b8ba2de9c710ea57a9c" } new_table_name: "default.Table_e012ec0c34264f4aab5eb939c1f7a34d" modify_external_catalogs: true
I20250625 13:52:13.656666 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_42544de6446c4aed9bddadf23288264f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.656839 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_520b3c352ef6433db536a0eca3be9231"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.657159 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42544de6446c4aed9bddadf23288264f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.657403 1168 raft_consensus.cc:2241] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:13.657557 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3c06b182c7854a5380dcb729131218d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.657687 1168 pending_rounds.cc:70] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:13.657737 1168 pending_rounds.cc:77] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520643208470528 op_type: NO_OP noop_request { }
I20250625 13:52:13.657829 1168 raft_consensus.cc:2887] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:13.657904 1168 raft_consensus.cc:2270] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:13.657927 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c06b182c7854a5380dcb729131218d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.658197 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_aeb56ffe1b1343528965b690ef571cc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.658447 1168 ts_tablet_manager.cc:1905] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:13.658564 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeb56ffe1b1343528965b690ef571cc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.659202 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_d0f7159508314d49992e34cf8af1e00c [id=a249a45df8b3454894e648d70f5e8b1f] alter complete (version 3)
I20250625 13:52:13.659474 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_60d3522fb58e4d2d8478eec6df099e91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.659869 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60d3522fb58e4d2d8478eec6df099e91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.661020 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf3168fb0cec45b5a45cd73eceee7414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.661283 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d0f8626d2ac470891919ef051d2bd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.667451 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f827dd12cc3049ba843bb47cb9e412d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.668391 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f1c1affc7c6a41e5b6d3bcb967d503da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.668928 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1c1affc7c6a41e5b6d3bcb967d503da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.671415 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2dafb49e9d4003962ebddd16a50f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.675542 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7779da6edee64fad8cbd0131510bebd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.677479 1166 tablet_bootstrap.cc:654] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:13.657166 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_520b3c352ef6433db536a0eca3be9231 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.619556 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf3168fb0cec45b5a45cd73eceee7414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.684120 2271 raft_consensus.cc:491] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.684243 2271 raft_consensus.cc:513] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.684895 2271 leader_election.cc:290] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:13.687289 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f827dd12cc3049ba843bb47cb9e412d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.632521 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 1d0f8626d2ac470891919ef051d2bd23 (DEFAULT_TABLE table=default.Table_a1144201285a4e62b6eedd9332730eb6 [id=74355b566c5f4e1aaf0883375522a419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.687860 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d0f8626d2ac470891919ef051d2bd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.640714 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d95a376304e464ab75e2e1e00e7a924 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.683089 2266 tablet.cc:1722] T dad4ea8cf44944b08f53428c95809daf P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:13.690536 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7779da6edee64fad8cbd0131510bebd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.632706 1022 tablet_service.cc:1468] Processing CreateTablet for tablet bf2dafb49e9d4003962ebddd16a50f76 (DEFAULT_TABLE table=default.Table_7222541aef894753bea095d602e31b74 [id=df2d817df2fc44e68b918280db6ebcdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.692178 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2dafb49e9d4003962ebddd16a50f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.697810 1014 tablet_service.cc:1468] Processing CreateTablet for tablet ee09a3942acc41acb4f7c0d5452bf268 (DEFAULT_TABLE table=default.Table_f4de7823c10e40e1bdd42b0e0559f7bf [id=7270b5aa9a0148178683a229db5b4b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.697837 879 tablet_service.cc:1468] Processing CreateTablet for tablet bf2dafb49e9d4003962ebddd16a50f76 (DEFAULT_TABLE table=default.Table_7222541aef894753bea095d602e31b74 [id=df2d817df2fc44e68b918280db6ebcdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.698180 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee09a3942acc41acb4f7c0d5452bf268. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.698230 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2dafb49e9d4003962ebddd16a50f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.699285 2348 raft_consensus.cc:491] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.699451 2348 raft_consensus.cc:513] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.701797 2348 leader_election.cc:290] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:13.702503 744 tablet_service.cc:1468] Processing CreateTablet for tablet 4efb4c779aae4126abba5563ef4330af (DEFAULT_TABLE table=default.Table_3ed19313833f4518bd076c28a982830a [id=566aa08c19274021b2d962037efe7c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.704239 882 tablet_service.cc:1468] Processing CreateTablet for tablet 1d0f8626d2ac470891919ef051d2bd23 (DEFAULT_TABLE table=default.Table_a1144201285a4e62b6eedd9332730eb6 [id=74355b566c5f4e1aaf0883375522a419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.704779 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d0f8626d2ac470891919ef051d2bd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.633690 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f24ae7c08bd42959ab48f7007ec98d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.628479 1170 ts_tablet_manager.cc:1397] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.297s user 0.012s sys 0.003s
I20250625 13:52:13.711252 1170 raft_consensus.cc:357] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.711447 1170 raft_consensus.cc:383] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.711494 1170 raft_consensus.cc:738] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.711687 1170 consensus_queue.cc:260] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.712193 1170 ts_tablet_manager.cc:1428] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:13.712447 1170 tablet_bootstrap.cc:492] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:13.712523 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c09eb0978274847b54e5665fdfae4f9 consensus state has no leader
I20250625 13:52:13.713997 886 tablet_service.cc:1468] Processing CreateTablet for tablet ee09a3942acc41acb4f7c0d5452bf268 (DEFAULT_TABLE table=default.Table_f4de7823c10e40e1bdd42b0e0559f7bf [id=7270b5aa9a0148178683a229db5b4b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.714092 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4efb4c779aae4126abba5563ef4330af (DEFAULT_TABLE table=default.Table_3ed19313833f4518bd076c28a982830a [id=566aa08c19274021b2d962037efe7c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.714391 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee09a3942acc41acb4f7c0d5452bf268. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.714390 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efb4c779aae4126abba5563ef4330af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.714568 1170 tablet_bootstrap.cc:654] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.715608 2313 raft_consensus.cc:491] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.715785 2313 raft_consensus.cc:513] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.717212 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efb4c779aae4126abba5563ef4330af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.718688 2313 leader_election.cc:290] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:13.683214 2345 raft_consensus.cc:491] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.720093 2345 raft_consensus.cc:513] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.723500 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:13.725517 711 leader_election.cc:343] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.727129 2345 leader_election.cc:290] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:13.728659 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d35517bbf1cb4695ae46094dc883175f consensus state has no leader
I20250625 13:52:13.732161 756 tablet_service.cc:1468] Processing CreateTablet for tablet ee09a3942acc41acb4f7c0d5452bf268 (DEFAULT_TABLE table=default.Table_f4de7823c10e40e1bdd42b0e0559f7bf [id=7270b5aa9a0148178683a229db5b4b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:13.732858 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee09a3942acc41acb4f7c0d5452bf268. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:13.742067 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b0b7c8da7ddc4130aa86a948c67edc38" } new_table_name: "default.Table_e154f1195be147aca7b2b5abad1d5f7c" modify_external_catalogs: true
W20250625 13:52:13.743661 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 56388.2us
W20250625 13:52:13.749032 851 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 6dd84b8afa2548fe8ce5925f0bf806f0
W20250625 13:52:13.749794 851 leader_election.cc:343] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a29708fb688a4a65b84ae2564a040b4c
I20250625 13:52:13.750996 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8a3bb463510942988dfd2cb114c96e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.751102 894 tablet_service.cc:1468] Processing CreateTablet for tablet 4efb4c779aae4126abba5563ef4330af (DEFAULT_TABLE table=default.Table_3ed19313833f4518bd076c28a982830a [id=566aa08c19274021b2d962037efe7c26]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:13.751544 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a3bb463510942988dfd2cb114c96e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.751569 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efb4c779aae4126abba5563ef4330af. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:13.753306 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e144ac4cd5e3415092856ff8e91016b3 consensus state has no leader
I20250625 13:52:13.753489 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8900946fe67c42d780eabc0aff65f01c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.754060 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8900946fe67c42d780eabc0aff65f01c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.748014 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:13.759195 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet ac4d368347444846880954ced9651404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.757282 2335 tablet.cc:1722] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250625 13:52:13.760203 711 leader_election.cc:343] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.760319 711 leader_election.cc:304] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:13.760749 2313 raft_consensus.cc:2747] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:13.761878 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b14d00c93a964edd815122e3723f89e7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:13.762051 913 raft_consensus.cc:2466] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:13.619285 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1055) took 1547 ms. Trace:
I20250625 13:52:13.762660 2254 rpcz_store.cc:276] 0625 13:52:12.071955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.072991 (+ 1036us) service_pool.cc:225] Handling call
0625 13:52:13.619182 (+1546191us) ts_tablet_manager.cc:1145] Deleting tablet 9302ac10a29c4b0c82268eb767615989
0625 13:52:13.619190 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.619270 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546054}
I20250625 13:52:13.766052 1168 ts_tablet_manager.cc:1918] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:13.766166 1168 log.cc:1199] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/067272869c5547648c90f83a34a036aa
I20250625 13:52:13.767336 1166 tablet_bootstrap.cc:492] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:13.767473 1166 ts_tablet_manager.cc:1397] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.155s user 0.004s sys 0.000s
I20250625 13:52:13.768229 1166 raft_consensus.cc:357] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.768389 1166 raft_consensus.cc:383] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.768436 1166 raft_consensus.cc:738] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:13.766224 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 145047us
W20250625 13:52:13.768972 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9302ac10a29c4b0c82268eb767615989 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9302ac10a29c4b0c82268eb767615989 already in progress: creating tablet
I20250625 13:52:13.767166 1170 tablet_bootstrap.cc:492] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:13.770496 1170 ts_tablet_manager.cc:1397] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.000s
I20250625 13:52:13.771143 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1049) took 1699 ms. Trace:
I20250625 13:52:13.771236 1170 raft_consensus.cc:357] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.772002 1170 raft_consensus.cc:383] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.772250 1170 raft_consensus.cc:738] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.772497 1170 consensus_queue.cc:260] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.773622 1170 ts_tablet_manager.cc:1428] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:13.771852 2254 rpcz_store.cc:276] 0625 13:52:12.071452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.072249 (+ 797us) service_pool.cc:225] Handling call
0625 13:52:13.769388 (+1697139us) ts_tablet_manager.cc:1145] Deleting tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7
0625 13:52:13.769402 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.771121 (+ 1719us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1696167}
W20250625 13:52:13.774335 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 already in progress: creating tablet
I20250625 13:52:13.776361 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1056) took 1704 ms. Trace:
I20250625 13:52:13.777034 2254 rpcz_store.cc:276] 0625 13:52:12.072041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.082213 (+ 10172us) service_pool.cc:225] Handling call
0625 13:52:13.775764 (+1693551us) ts_tablet_manager.cc:1145] Deleting tablet bdc5395c1de349b9b20d28625b8751f7
0625 13:52:13.775776 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.776190 (+ 414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693201}
W20250625 13:52:13.778409 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bdc5395c1de349b9b20d28625b8751f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdc5395c1de349b9b20d28625b8751f7 already in progress: creating tablet
I20250625 13:52:13.779577 2254 tablet_replica.cc:331] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:13.780907 1170 tablet_bootstrap.cc:492] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:13.781152 2254 raft_consensus.cc:2241] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.782589 1170 tablet_bootstrap.cc:654] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.782200 2254 raft_consensus.cc:2270] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.783842 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:13.786306 2300 raft_consensus.cc:491] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.786458 2300 raft_consensus.cc:513] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.786953 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 38a23b08a03b48eaa051f670eb7dac65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.787237 2300 leader_election.cc:290] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:13.789163 2254 ts_tablet_manager.cc:1905] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.791899 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:52:13.792637 985 leader_election.cc:343] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: df1ed922750f45eab02d1fc0c26af75e
I20250625 13:52:13.793949 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f85b46b8fe3f4f589471d8c43630f7ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.794461 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f85b46b8fe3f4f589471d8c43630f7ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.796504 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a] alter complete (version 1)
I20250625 13:52:13.796450 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_66528a87070643a69c4220dd80f34de0" } new_table_name: "default.Table_54d47be8fb054bcfa11587b4eb3ef955" modify_external_catalogs: true
I20250625 13:52:13.804147 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
W20250625 13:52:13.804500 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c450ae4c434e4c6fa73d06c184b2bdc7 consensus state has no leader
I20250625 13:52:13.809067 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f37637a5cbf94eb986a689ea87854c41" } modify_external_catalogs: true
I20250625 13:52:13.809355 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f37637a5cbf94eb986a689ea87854c41" } modify_external_catalogs: true
I20250625 13:52:13.809775 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_74dfe85879a04d7eb5bee2e8f00065ab" } new_table_name: "default.Table_2498f4d6086d4a1b98304771c890d55f" modify_external_catalogs: true
I20250625 13:52:13.811636 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e012ec0c34264f4aab5eb939c1f7a34d" } modify_external_catalogs: true
I20250625 13:52:13.811787 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e012ec0c34264f4aab5eb939c1f7a34d" } modify_external_catalogs: true
I20250625 13:52:13.814205 1168 ts_tablet_manager.cc:1939] T 067272869c5547648c90f83a34a036aa P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:13.814908 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f7f814de767d41a4b070ac5045680405" } modify_external_catalogs: true
I20250625 13:52:13.815047 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f7f814de767d41a4b070ac5045680405" } modify_external_catalogs: true
I20250625 13:52:13.815071 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 874) took 4178 ms. Trace:
I20250625 13:52:13.814978 1166 consensus_queue.cc:260] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.815596 1166 ts_tablet_manager.cc:1428] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20250625 13:52:13.815150 1168 rpcz_store.cc:276] 0625 13:52:09.636812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.636873 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:13.595796 (+3958923us) ts_tablet_manager.cc:1145] Deleting tablet 067272869c5547648c90f83a34a036aa
0625 13:52:13.595804 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.753135 (+157331us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.765957 (+ 12822us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.815047 (+ 49090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3958784}
I20250625 13:52:13.817256 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_51a539bc437b40f6ae158307f8306a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.817512 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b120e9ea0fc04b3dbce52cdc33a713d5 consensus state has no leader
W20250625 13:52:13.817754 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51a539bc437b40f6ae158307f8306a0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.818362 1168 tablet_replica.cc:331] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:13.818598 1168 raft_consensus.cc:2241] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.818848 1168 raft_consensus.cc:2270] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:13.819543 849 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.009s sys 0.010s
I20250625 13:52:13.819607 1165 ts_tablet_manager.cc:1918] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:13.819666 849 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.009s sys 0.010s
W20250625 13:52:13.819737 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 555079us
I20250625 13:52:13.820488 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d35517bbf1cb4695ae46094dc883175f
I20250625 13:52:13.821174 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a29708fb688a4a65b84ae2564a040b4c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:13.822486 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9514bbbda0ba4329a18b0a42e635c6e5" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:13.824227 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 55a254d82f764aa2860e465b4857a413
I20250625 13:52:13.825068 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 55a254d82f764aa2860e465b4857a413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.825248 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79beb23eebf54fd9951ea5a33fcfb0ae" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:13.825943 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6c331c57bff44091be2849c21d6ab2d7
I20250625 13:52:13.826880 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c331c57bff44091be2849c21d6ab2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.827394 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c331c57bff44091be2849c21d6ab2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.827669 595 master_service.cc:948] ReplaceTablet: received request to replace tablet dc1c9cea3d5a405a958c6c7585dbef39 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:13.827731 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d35517bbf1cb4695ae46094dc883175f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.825204 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 55a254d82f764aa2860e465b4857a413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.826022 1166 tablet_bootstrap.cc:492] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:13.826467 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c331c57bff44091be2849c21d6ab2d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.829172 1165 log.cc:1199] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/60ef50a3624b4c79acc803ff0e2ed364
I20250625 13:52:13.829548 1165 ts_tablet_manager.cc:1939] T 60ef50a3624b4c79acc803ff0e2ed364 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:13.832278 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c9c75aecd94048888373fbb8cc3efdbd" } modify_external_catalogs: true
I20250625 13:52:13.832458 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c9c75aecd94048888373fbb8cc3efdbd" } modify_external_catalogs: true
I20250625 13:52:13.834220 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 55a254d82f764aa2860e465b4857a413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.840147 1171 tablet_bootstrap.cc:492] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:13.840348 1171 ts_tablet_manager.cc:1397] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.329s user 0.015s sys 0.001s
I20250625 13:52:13.841149 1171 raft_consensus.cc:357] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.841394 1171 raft_consensus.cc:383] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.841502 1171 raft_consensus.cc:738] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.841743 1171 consensus_queue.cc:260] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.842404 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6fcc17f839504d6b85526b638763ec5b
I20250625 13:52:13.842449 1171 ts_tablet_manager.cc:1428] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:13.842839 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df1ed922750f45eab02d1fc0c26af75e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:13.843017 912 raft_consensus.cc:2466] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:13.843209 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fcc17f839504d6b85526b638763ec5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.827942 756 tablet_service.cc:1515] Processing DeleteTablet for tablet d35517bbf1cb4695ae46094dc883175f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.843551 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab7202a8761444a0a731993d890fcdeb" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:13.843766 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fcc17f839504d6b85526b638763ec5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.844374 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fcc17f839504d6b85526b638763ec5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.849602 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4293c9800b3c40498bd0d764bf20c537" } modify_external_catalogs: true
I20250625 13:52:13.849830 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4293c9800b3c40498bd0d764bf20c537" } modify_external_catalogs: true
I20250625 13:52:13.852491 882 tablet_service.cc:1515] Processing DeleteTablet for tablet d35517bbf1cb4695ae46094dc883175f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.854285 2322 tablet.cc:1722] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:13.855683 983 leader_election.cc:304] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:13.856081 2300 raft_consensus.cc:2802] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:13.856163 2300 raft_consensus.cc:491] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:13.856222 2300 raft_consensus.cc:3058] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:13.857627 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a7b72345e04445f7a393c469a348d126
I20250625 13:52:13.858337 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a7b72345e04445f7a393c469a348d126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.859014 2300 raft_consensus.cc:513] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.859340 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a7b72345e04445f7a393c469a348d126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.859967 2313 raft_consensus.cc:491] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.860098 2313 raft_consensus.cc:513] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.860669 2300 leader_election.cc:290] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:13.861258 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a7b72345e04445f7a393c469a348d126 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:13 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:13.861641 985 leader_election.cc:343] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b14d00c93a964edd815122e3723f89e7
I20250625 13:52:13.861606 849 leader_election.cc:304] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:13.864451 2204 raft_consensus.cc:2802] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:13.864717 2204 raft_consensus.cc:695] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:13.865044 2204 consensus_queue.cc:237] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:13.872393 849 leader_election.cc:343] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 6dd84b8afa2548fe8ce5925f0bf806f0
I20250625 13:52:13.872570 849 leader_election.cc:304] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:13.872972 849 raft_consensus.cc:2684] T 6dd84b8afa2548fe8ce5925f0bf806f0 P affa1c3035b940e88ca143be7d35840d: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 13:52:13.874284 849 leader_election.cc:343] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.874462 849 leader_election.cc:304] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:13.874718 2210 raft_consensus.cc:2747] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:13.875110 849 leader_election.cc:343] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.875439 2210 consensus_queue.cc:1035] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:13.885241 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 156da2a965c64d1090a4f9d4eafdec70 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:13.886088 983 leader_election.cc:304] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:13.886898 2300 raft_consensus.cc:2802] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:13.886987 2300 raft_consensus.cc:491] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:13.887048 2300 raft_consensus.cc:3058] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:13.889782 2300 raft_consensus.cc:513] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.890717 2300 leader_election.cc:290] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:13.891356 985 leader_election.cc:343] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: df1ed922750f45eab02d1fc0c26af75e
I20250625 13:52:13.891417 1168 ts_tablet_manager.cc:1905] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.893410 1166 tablet_bootstrap.cc:654] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.894595 1165 tablet_replica.cc:331] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:13.894807 1165 raft_consensus.cc:2241] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.894956 1165 raft_consensus.cc:2270] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.895466 1165 ts_tablet_manager.cc:1905] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.897415 2313 leader_election.cc:290] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:13.898597 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da2f8d0d9a714a28ba6fb9642d461de2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:13.899166 711 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.903188 1165 ts_tablet_manager.cc:1918] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.906401 1165 log.cc:1199] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/69672579a83c41d09ee7c78a6fa7d746
I20250625 13:52:13.906562 2254 ts_tablet_manager.cc:1918] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:13.907495 1165 ts_tablet_manager.cc:1939] T 69672579a83c41d09ee7c78a6fa7d746 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:13.907367 2254 log.cc:1199] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/5cf73b4ab99c466abcfedf1b200e743b
I20250625 13:52:13.909340 983 leader_election.cc:304] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:13.909835 2300 raft_consensus.cc:2747] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:13.910684 2254 ts_tablet_manager.cc:1939] T 5cf73b4ab99c466abcfedf1b200e743b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:13.912580 1165 tablet_replica.cc:331] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:13.912813 1165 raft_consensus.cc:2241] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.913165 1165 raft_consensus.cc:2270] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:13.913232 851 leader_election.cc:343] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9514bbbda0ba4329a18b0a42e635c6e5
I20250625 13:52:13.913424 851 leader_election.cc:304] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:13.913676 2210 raft_consensus.cc:491] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.913803 2210 raft_consensus.cc:513] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.914372 2210 leader_election.cc:290] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:13.914502 2210 raft_consensus.cc:2747] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:13.914387 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1058) took 1831 ms. Trace:
I20250625 13:52:13.915794 1166 tablet_bootstrap.cc:492] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:13.915952 1166 ts_tablet_manager.cc:1397] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.001s
I20250625 13:52:13.915141 2254 rpcz_store.cc:276] 0625 13:52:12.083153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.083202 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:13.779209 (+1696007us) ts_tablet_manager.cc:1145] Deleting tablet 5cf73b4ab99c466abcfedf1b200e743b
0625 13:52:13.779222 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:13.872272 (+ 93050us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.905787 (+ 33515us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:13.914364 (+ 8577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1695666}
I20250625 13:52:13.916676 1166 raft_consensus.cc:357] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.916752 2335 tablet.cc:1717] T 6e529f5bf13f4ee69e66402b1ee252d4 P affa1c3035b940e88ca143be7d35840d: Skipping requested alter to schema version 2, tablet already version 2
I20250625 13:52:13.916898 1166 raft_consensus.cc:383] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.917029 1166 raft_consensus.cc:738] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.917032 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 5cf73b4ab99c466abcfedf1b200e743b (table default.Table_47598e6b1611413784a8c9ab9d1c0a67 [id=8c0cf5af3aee43e9b7676c1be4842093]) successfully deleted
I20250625 13:52:13.917114 2335 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52994 (request call id 1104) took 1078 ms. Trace:
I20250625 13:52:13.917300 1166 consensus_queue.cc:260] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:13.917182 2335 rpcz_store.cc:276] 0625 13:52:12.838323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.838990 (+ 667us) service_pool.cc:225] Handling call
0625 13:52:13.917101 (+1078111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 13:52:13.107817 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0625 13:52:13.107919 (+ 102us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0625 13:52:13.338229 (+230310us) alter_schema_op.cc:66] Acquired schema lock
0625 13:52:13.338240 (+ 11us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0625 13:52:13.338288 (+ 48us) alter_schema_op.cc:128] START. Timestamp: 1750859533338248
0625 13:52:13.338293 (+ 5us) op_driver.cc:348] REPLICATION: starting
0625 13:52:13.537292 (+198999us) log.cc:844] Serialized 173 byte log entry
0625 13:52:13.545355 (+ 8063us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x616000095554. stack: 00007f37024dc336 00007f37024dc0f0 00007f37014d23ad 0000000000971004 0000000000970fcd 000000000097099a 00007f370df11762 00007f370f2b8855 00007f370f2b7379 00007f370f2bf7a0 00007f370f2bf5ad 0000000000a7573c 00007f37025219df 00007f3702524cab 00007f3702524b28 0000000000a7573c
0625 13:52:13.916321 (+370966us) op_driver.cc:464] REPLICATION: finished
0625 13:52:13.916724 (+ 403us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0625 13:52:13.916962 (+ 238us) log.cc:844] Serialized 94 byte log entry
0625 13:52:13.917001 (+ 39us) alter_schema_op.cc:73] Released schema lock
0625 13:52:13.917065 (+ 64us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":344,"prepare.queue_time_us":167709,"prepare.run_cpu_time_us":12682,"prepare.run_wall_time_us":437565,"replication_time_us":577996,"spinlock_wait_cycles":457237888}]]}
I20250625 13:52:13.918558 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_232257f7ad2d4eb689c206dfde0510fd" } new_table_name: "default.Table_2c311e79aef2487d8524720a4cd0677c" modify_external_catalogs: true
I20250625 13:52:13.918252 2254 tablet_replica.cc:331] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:13.917766 1166 ts_tablet_manager.cc:1428] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:13.921190 1166 tablet_bootstrap.cc:492] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:13.920825 2254 raft_consensus.cc:2241] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:13.922185 2254 raft_consensus.cc:2270] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:13.923429 1166 tablet_bootstrap.cc:654] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:13.924319 1170 tablet_bootstrap.cc:492] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:13.924505 1170 ts_tablet_manager.cc:1397] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.144s user 0.009s sys 0.007s
I20250625 13:52:13.925216 1170 raft_consensus.cc:357] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.926016 1170 raft_consensus.cc:383] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:13.926126 1170 raft_consensus.cc:738] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:13.926606 1170 consensus_queue.cc:260] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.927248 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 60ef50a3624b4c79acc803ff0e2ed364 (table default.Table_3e7baa9cc87143b0833a6bc32a193408 [id=d043c74a6cb94efa89ce819d2ef35545]) successfully deleted
W20250625 13:52:13.927544 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 569f6cba69394f2fa005686056f19929 because the tablet was not found. No further retry: Not found: Tablet not found: 569f6cba69394f2fa005686056f19929
I20250625 13:52:13.927701 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 69672579a83c41d09ee7c78a6fa7d746 (table default.Table_531f237d57114b2483ae3cf8d4b94659 [id=a525a4b043274f8ea88049ed7d1cf652]) successfully deleted
I20250625 13:52:13.928858 1165 ts_tablet_manager.cc:1905] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:13.930440 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da2f8d0d9a714a28ba6fb9642d461de2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:13.930804 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df1ed922750f45eab02d1fc0c26af75e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:13.930943 916 raft_consensus.cc:3058] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:13.933501 756 tablet_service.cc:1515] Processing DeleteTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.933691 916 raft_consensus.cc:2466] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:13.933732 753 tablet_service.cc:1515] Processing DeleteTablet for tablet b7d888e628a94f939ecf2571e60b1948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:13.934003 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b14d00c93a964edd815122e3723f89e7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:13.934136 914 raft_consensus.cc:3058] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:13.936761 914 raft_consensus.cc:2466] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:13.939759 711 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:13.939879 711 leader_election.cc:304] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:13.940088 2313 raft_consensus.cc:2747] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:13.941004 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6dc8600ee239471dbe4e422ce34db986"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.941051 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_28183b013d3c41b4a66d5037fd020e5b" } new_table_name: "default.Table_262ec080b91d44d68afd7d9c6341cb25" modify_external_catalogs: true
I20250625 13:52:13.941403 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_45c7b1313316471f9fc8a44c2f19554b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.941578 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dc8600ee239471dbe4e422ce34db986 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:13.941869 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45c7b1313316471f9fc8a44c2f19554b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.942003 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_177c0af3b570495e804887f8eff6a8d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.930516 2254 ts_tablet_manager.cc:1905] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:13.942416 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_177c0af3b570495e804887f8eff6a8d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.941881 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5e8ec384f9c44f70a2494253304496f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.945801 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e8ec384f9c44f70a2494253304496f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.948453 2266 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250625 13:52:13.948597 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d34a4de076842a5966df1dc28548a4c has no consensus state
I20250625 13:52:13.949427 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7ce4c37b46af4972a39f3bd6c431af1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.950032 2322 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250625 13:52:13.951756 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ce4c37b46af4972a39f3bd6c431af1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.952162 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9c3c0c15b51d4dca8c18dfb78f2054d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.952605 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c3c0c15b51d4dca8c18dfb78f2054d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.952805 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:13.954572 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f5c726e169ca41b6a7092eef2e923eb5" } new_table_name: "default.Table_98a06566efb2452d88ce58813730c271" modify_external_catalogs: true
I20250625 13:52:13.958220 2348 raft_consensus.cc:491] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:13.958365 2348 raft_consensus.cc:513] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:13.958827 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2b9e44578bd84aa6a63b1371d17d2318"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:13.958992 2348 leader_election.cc:290] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:13.959291 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b9e44578bd84aa6a63b1371d17d2318 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:13.960805 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_652e0ff5dc3f4eb286b425ec0630c138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:13.961326 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_652e0ff5dc3f4eb286b425ec0630c138 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:13.962916 985 leader_election.cc:343] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2404e712e18146b68757bd77362c1358
W20250625 13:52:13.969749 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 47638.7us
I20250625 13:52:13.937004 1170 ts_tablet_manager.cc:1428] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250625 13:52:13.972512 1171 tablet_bootstrap.cc:492] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:13.974987 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18014e8cb05a44e694e9c36d7a2aecbc consensus state has no leader
I20250625 13:52:13.979703 597 master_service.cc:948] ReplaceTablet: received request to replace tablet d7f68a0c4bb341a9a90cc2be0f78daca from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:13.980991 1170 tablet_bootstrap.cc:492] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:13.983595 592 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_2c311e79aef2487d8524720a4cd0677c [id=71729b0c8f7046bb8e80bb96f9f2ba71] alter complete (version 4)
I20250625 13:52:13.992517 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 87f5b36ddb474ed6903c14876b236ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet db6e9db2375144b38ec43ffca9ca4c1e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:13.992532 2335 tablet.cc:1722] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:14.004675 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_78e65c78bd4d43f0900060ea0aa8203a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.006105 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_def99fb8fc1f4cdd86436da682cdae13" } new_table_name: "default.Table_8315e81e0d0843819735795f79fa91fb" modify_external_catalogs: true
W20250625 13:52:14.006929 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78e65c78bd4d43f0900060ea0aa8203a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.008386 1170 tablet_bootstrap.cc:654] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:14.011620 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 95584.7us
W20250625 13:52:14.012251 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 669de0b7c93a4efea8623dab2e462c8d consensus state has no leader
W20250625 13:52:14.012645 851 consensus_peers.cc:487] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77c50e1ebb9042ef8f0a651ba7186b74. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:14.013608 1044 raft_consensus.cc:3058] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.018921 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fd92384a196c4ca1a02f43763f639a77" } new_table_name: "default.Table_5e22544eb55d4668b8b7231c09b6adb2" modify_external_catalogs: true
I20250625 13:52:14.022399 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c5d44351d1f8484294462de887c97b18" } modify_external_catalogs: true
I20250625 13:52:14.022606 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c5d44351d1f8484294462de887c97b18" } modify_external_catalogs: true
W20250625 13:52:14.024133 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af497b011b7248d2b247854b150a8f21 consensus state has no leader
I20250625 13:52:14.025631 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e2e20a31ea3544cf974a8487d18364e7" } modify_external_catalogs: true
I20250625 13:52:14.025877 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e2e20a31ea3544cf974a8487d18364e7" } modify_external_catalogs: true
I20250625 13:52:14.027644 2266 tablet.cc:1722] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:14.028271 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4c2981fcc6824fd4aa3f0c4eb5b82d6c" } modify_external_catalogs: true
I20250625 13:52:14.028405 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4c2981fcc6824fd4aa3f0c4eb5b82d6c" } modify_external_catalogs: true
I20250625 13:52:14.039454 595 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]). Expected version 1 got 0
I20250625 13:52:14.044023 1165 ts_tablet_manager.cc:1918] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.045925 2254 ts_tablet_manager.cc:1918] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.046906 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_76340fa76a5048f9b56482e7e2563551" } modify_external_catalogs: true
I20250625 13:52:14.047174 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_76340fa76a5048f9b56482e7e2563551" } modify_external_catalogs: true
I20250625 13:52:14.047590 2335 tablet.cc:1722] T dad4ea8cf44944b08f53428c95809daf P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:14.047752 1165 log.cc:1199] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:52:14.047876 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_42800fa616864465b16c7cb54ad50494"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.012310 1171 tablet_bootstrap.cc:654] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.046710 1170 tablet_bootstrap.cc:492] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:14.049189 1170 ts_tablet_manager.cc:1397] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.001s
I20250625 13:52:14.050030 1170 raft_consensus.cc:357] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.050258 1170 raft_consensus.cc:383] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.050359 1170 raft_consensus.cc:738] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.050644 1170 consensus_queue.cc:260] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.051138 1170 ts_tablet_manager.cc:1428] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:14.051469 1170 tablet_bootstrap.cc:492] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:14.052943 1170 tablet_bootstrap.cc:654] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.054221 2254 log.cc:1199] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/77c50e1ebb9042ef8f0a651ba7186b74
I20250625 13:52:14.054597 1165 ts_tablet_manager.cc:1939] T c605d5b64c164c1594b0a40cfa2c4cf6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:14.056594 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d43ba17ebfec4244894d1b6e23200c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.057502 2254 ts_tablet_manager.cc:1939] T 77c50e1ebb9042ef8f0a651ba7186b74 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:14.060683 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1057) took 1977 ms. Trace:
W20250625 13:52:14.061863 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42800fa616864465b16c7cb54ad50494 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.061496 2254 rpcz_store.cc:276] 0625 13:52:12.083030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.099406 (+ 16376us) service_pool.cc:225] Handling call
0625 13:52:13.917976 (+1818570us) ts_tablet_manager.cc:1145] Deleting tablet 77c50e1ebb9042ef8f0a651ba7186b74
0625 13:52:13.917988 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.012964 (+ 94976us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.045827 (+ 32863us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.060662 (+ 14835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818118}
I20250625 13:52:14.064196 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cbc60d9e09c5478391309eae3867f656" } modify_external_catalogs: true
I20250625 13:52:14.064390 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cbc60d9e09c5478391309eae3867f656" } modify_external_catalogs: true
I20250625 13:52:14.070127 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet c605d5b64c164c1594b0a40cfa2c4cf6 (table default.Table_1956bf8628234a2fa1b29b76594f624b [id=d3d911b4c96949afa23f4f64e9b5bbd0]) successfully deleted
W20250625 13:52:14.070856 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 20841d3424fb4c48b4de4ed7b2917abe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20841d3424fb4c48b4de4ed7b2917abe already in progress: creating tablet
W20250625 13:52:14.071449 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9f24ae7c08bd42959ab48f7007ec98d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f24ae7c08bd42959ab48f7007ec98d3 already in progress: creating tablet
I20250625 13:52:14.071926 1165 tablet_replica.cc:331] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:14.072225 2335 tablet.cc:1722] T 79beb23eebf54fd9951ea5a33fcfb0ae P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:14.072281 1165 raft_consensus.cc:2241] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.072638 1165 raft_consensus.cc:2270] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.073619 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 77c50e1ebb9042ef8f0a651ba7186b74 (table default.Table_bdcf17e5612a44579ba27af546ca1def [id=cc778fd879634dd6a2ba7ba8274c9ef4]) successfully deleted
I20250625 13:52:14.074000 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1068) took 1850 ms. Trace:
I20250625 13:52:14.074128 2254 rpcz_store.cc:276] 0625 13:52:12.223654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.224464 (+ 810us) service_pool.cc:225] Handling call
0625 13:52:14.073894 (+1849430us) ts_tablet_manager.cc:1145] Deleting tablet 285912ccbf5a4e5bab53c48d1c024bb8
0625 13:52:14.073904 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.073982 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849226}
W20250625 13:52:14.074616 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 285912ccbf5a4e5bab53c48d1c024bb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 285912ccbf5a4e5bab53c48d1c024bb8 already in progress: creating tablet
I20250625 13:52:14.074968 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1069) took 1851 ms. Trace:
I20250625 13:52:14.075078 2254 rpcz_store.cc:276] 0625 13:52:12.223743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.241812 (+ 18069us) service_pool.cc:225] Handling call
0625 13:52:14.074874 (+1833062us) ts_tablet_manager.cc:1145] Deleting tablet 4111b95923ed4d73b7eca60b964d9494
0625 13:52:14.074883 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.074952 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832899}
W20250625 13:52:14.075656 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 4111b95923ed4d73b7eca60b964d9494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4111b95923ed4d73b7eca60b964d9494 already in progress: creating tablet
I20250625 13:52:14.076122 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1071) took 1852 ms. Trace:
I20250625 13:52:14.076279 2254 rpcz_store.cc:276] 0625 13:52:12.223905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.252295 (+ 28390us) service_pool.cc:225] Handling call
0625 13:52:14.075951 (+1823656us) ts_tablet_manager.cc:1145] Deleting tablet 6110bcce59ea452cbc6e9f464483bde9
0625 13:52:14.075960 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.076083 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1823450}
W20250625 13:52:14.076838 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6110bcce59ea452cbc6e9f464483bde9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6110bcce59ea452cbc6e9f464483bde9 already in progress: creating tablet
I20250625 13:52:14.077253 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1072) took 1853 ms. Trace:
I20250625 13:52:14.086812 2366 tablet.cc:1722] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:14.077423 2254 rpcz_store.cc:276] 0625 13:52:12.223966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.252535 (+ 28569us) service_pool.cc:225] Handling call
0625 13:52:14.077116 (+1824581us) ts_tablet_manager.cc:1145] Deleting tablet ea383a1ceddc461fb2531dbca714029e
0625 13:52:14.077124 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.077239 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1779565}
W20250625 13:52:14.091511 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ea383a1ceddc461fb2531dbca714029e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea383a1ceddc461fb2531dbca714029e already in progress: creating tablet
I20250625 13:52:14.093104 1165 ts_tablet_manager.cc:1905] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:14.094313 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1073) took 1870 ms. Trace:
I20250625 13:52:14.095348 2254 rpcz_store.cc:276] 0625 13:52:12.224027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.255206 (+ 31179us) service_pool.cc:225] Handling call
0625 13:52:14.092905 (+1837699us) ts_tablet_manager.cc:1145] Deleting tablet c27caba6c4d14d51b10bf04f131c6a3c
0625 13:52:14.093168 (+ 263us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.094059 (+ 891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795030}
W20250625 13:52:14.097446 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet c27caba6c4d14d51b10bf04f131c6a3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c27caba6c4d14d51b10bf04f131c6a3c already in progress: creating tablet
I20250625 13:52:14.099462 2266 tablet.cc:1717] T 6e529f5bf13f4ee69e66402b1ee252d4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Skipping requested alter to schema version 2, tablet already version 2
I20250625 13:52:14.102838 983 leader_election.cc:304] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.103358 2300 raft_consensus.cc:2802] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:14.103672 2300 raft_consensus.cc:695] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:14.103999 2300 consensus_queue.cc:237] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:14.105867 983 leader_election.cc:304] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.106266 2300 raft_consensus.cc:2802] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:14.106379 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1089) took 1676 ms. Trace:
I20250625 13:52:14.112797 2322 tablet.cc:1717] T 6e529f5bf13f4ee69e66402b1ee252d4 P 5c1aa07ab6bc4b0089e46951240a37c9: Skipping requested alter to schema version 2, tablet already version 2
I20250625 13:52:14.106490 2300 raft_consensus.cc:695] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:14.113243 2300 consensus_queue.cc:237] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:14.115999 2322 tablet.cc:1722] T 86af0ea6648045a1ac4ba903d20849b7 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:14.119907 2348 raft_consensus.cc:491] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.120050 2348 raft_consensus.cc:513] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.120764 2348 leader_election.cc:290] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.121421 2254 rpcz_store.cc:276] 0625 13:52:12.429670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.429843 (+ 173us) service_pool.cc:225] Handling call
0625 13:52:14.106257 (+1676414us) ts_tablet_manager.cc:1145] Deleting tablet a8c4f90b40c344168e8f689c43101a64
0625 13:52:14.106267 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.106363 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676092}
W20250625 13:52:14.121881 985 leader_election.cc:343] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ac4d368347444846880954ced9651404
W20250625 13:52:14.121984 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a8c4f90b40c344168e8f689c43101a64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c4f90b40c344168e8f689c43101a64 already in progress: creating tablet
I20250625 13:52:14.122381 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1087) took 1692 ms. Trace:
I20250625 13:52:14.128558 2254 rpcz_store.cc:276] 0625 13:52:12.429464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.434776 (+ 5312us) service_pool.cc:225] Handling call
0625 13:52:14.122258 (+1687482us) ts_tablet_manager.cc:1145] Deleting tablet 30b5b3444fd245bbb2ecd5889d956862
0625 13:52:14.122269 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.122365 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1686310}
I20250625 13:52:14.129319 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 19f1050a29b540558aaf1e31118e4531
I20250625 13:52:14.129344 2348 raft_consensus.cc:491] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.129477 2348 raft_consensus.cc:513] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.129770 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1086) took 1700 ms. Trace:
I20250625 13:52:14.130069 2348 leader_election.cc:290] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:14.130115 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 30b5b3444fd245bbb2ecd5889d956862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30b5b3444fd245bbb2ecd5889d956862 already in progress: creating tablet
I20250625 13:52:14.130661 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 19f1050a29b540558aaf1e31118e4531 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.130851 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 19f1050a29b540558aaf1e31118e4531 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:14.131078 985 leader_election.cc:343] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 38a23b08a03b48eaa051f670eb7dac65
I20250625 13:52:14.132707 2322 tablet.cc:1722] T 79beb23eebf54fd9951ea5a33fcfb0ae P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:14.133109 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1f47260bf0ad41fdbfbad50380a12547
I20250625 13:52:14.133941 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f47260bf0ad41fdbfbad50380a12547 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.134943 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f47260bf0ad41fdbfbad50380a12547 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.129839 2254 rpcz_store.cc:276] 0625 13:52:12.429388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.435993 (+ 6605us) service_pool.cc:225] Handling call
0625 13:52:14.129647 (+1693654us) ts_tablet_manager.cc:1145] Deleting tablet 05364e23e0ec4cffb92a8df2d38ca4b9
0625 13:52:14.129658 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.129755 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693447}
W20250625 13:52:14.135633 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05364e23e0ec4cffb92a8df2d38ca4b9 already in progress: creating tablet
I20250625 13:52:14.136171 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1085) took 1706 ms. Trace:
I20250625 13:52:14.136348 1166 tablet_bootstrap.cc:492] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:14.136554 1166 ts_tablet_manager.cc:1397] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.215s user 0.009s sys 0.007s
I20250625 13:52:14.137377 1166 raft_consensus.cc:357] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.137609 1166 raft_consensus.cc:383] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.137712 1166 raft_consensus.cc:738] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.137948 1166 consensus_queue.cc:260] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:14.138971 849 consensus_peers.cc:487] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:14.139096 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2b29392cfeb14f46a60fd0731f220127
I20250625 13:52:14.140270 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b29392cfeb14f46a60fd0731f220127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.140750 590 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_fac1b7cf47334e72b5815c78f0e35ba6 [id=65ec4be481574b20a64c98a75c6c4367] alter complete (version 4)
I20250625 13:52:14.140775 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 19f1050a29b540558aaf1e31118e4531 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.136305 2254 rpcz_store.cc:276] 0625 13:52:12.429316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.436232 (+ 6916us) service_pool.cc:225] Handling call
0625 13:52:14.136054 (+1699822us) ts_tablet_manager.cc:1145] Deleting tablet b70e7607709e4be395f1ed8d01d7d169
0625 13:52:14.136064 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.136155 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1699644}
I20250625 13:52:14.141360 1166 ts_tablet_manager.cc:1428] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20250625 13:52:14.141423 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet b70e7607709e4be395f1ed8d01d7d169 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b70e7607709e4be395f1ed8d01d7d169 already in progress: creating tablet
I20250625 13:52:14.141602 1166 tablet_bootstrap.cc:492] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:14.141961 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1091) took 1703 ms. Trace:
I20250625 13:52:14.143085 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f7ba037477124796ab2a66a351968c19" } new_table_name: "default.Table_03beb73b975f4565b8cecb57cc54e14d" modify_external_catalogs: true
I20250625 13:52:14.143460 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f47260bf0ad41fdbfbad50380a12547 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.144956 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc
I20250625 13:52:14.146106 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.146116 2254 rpcz_store.cc:276] 0625 13:52:12.438901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.438970 (+ 69us) service_pool.cc:225] Handling call
0625 13:52:14.141763 (+1702793us) ts_tablet_manager.cc:1145] Deleting tablet 34371795c8094e73ab4e5dbeedb7aa2f
0625 13:52:14.141775 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.141919 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702512}
I20250625 13:52:14.140532 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b29392cfeb14f46a60fd0731f220127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.146168 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:14.146763 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 34371795c8094e73ab4e5dbeedb7aa2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34371795c8094e73ab4e5dbeedb7aa2f already in progress: creating tablet
I20250625 13:52:14.147397 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b29392cfeb14f46a60fd0731f220127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.147257 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1090) took 1717 ms. Trace:
I20250625 13:52:14.147804 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.151147 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1e3b96b3cabb46eea9cb5b0275bcda73" } new_table_name: "default.Table_9288bcd2d4bc4fdf90ea3fc0f27be760" modify_external_catalogs: true
I20250625 13:52:14.152886 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a30a35e3639a43c5b078169060824723 with delete_type TABLET_DATA_DELETED (replaced by tablet efc9536a716f4b9f9fc4af1637b3cd80) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:14.153247 849 leader_election.cc:343] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:14.154217 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_22c63d145f8844b286282fdf48f6c32c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.154721 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22c63d145f8844b286282fdf48f6c32c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:14.158349 849 consensus_peers.cc:487] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:14.159222 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38a23b08a03b48eaa051f670eb7dac65" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:14.159461 914 raft_consensus.cc:2466] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:14.159969 983 leader_election.cc:304] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:14.160167 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d7f68a0c4bb341a9a90cc2be0f78daca
I20250625 13:52:14.160444 2348 raft_consensus.cc:2747] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.160601 597 catalog_manager.cc:6652] ReplaceTablet: tablet d7f68a0c4bb341a9a90cc2be0f78daca deleted and replaced by tablet 0c49eab755dd4f1cb16da4e10a24eecb
I20250625 13:52:14.160934 756 tablet_service.cc:1515] Processing DeleteTablet for tablet d7f68a0c4bb341a9a90cc2be0f78daca with delete_type TABLET_DATA_DELETED (replaced by tablet 0c49eab755dd4f1cb16da4e10a24eecb) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.161275 983 leader_election.cc:304] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:14.161569 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf6c3973912b4c5fbec28b60c4fdb408 consensus state has no leader
I20250625 13:52:14.161273 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d7f68a0c4bb341a9a90cc2be0f78daca with delete_type TABLET_DATA_DELETED (replaced by tablet 0c49eab755dd4f1cb16da4e10a24eecb) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.162178 2348 raft_consensus.cc:2802] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:14.162272 2348 raft_consensus.cc:491] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:14.162346 2348 raft_consensus.cc:3058] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.147843 2254 rpcz_store.cc:276] 0625 13:52:12.429780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.435726 (+ 5946us) service_pool.cc:225] Handling call
0625 13:52:14.147069 (+1711343us) ts_tablet_manager.cc:1145] Deleting tablet e438e46874f841d2ac972b953ddb4608
0625 13:52:14.147103 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.147240 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1703539}
W20250625 13:52:14.163215 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e438e46874f841d2ac972b953ddb4608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e438e46874f841d2ac972b953ddb4608 already in progress: creating tablet
I20250625 13:52:14.163682 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1092) took 1630 ms. Trace:
W20250625 13:52:14.164248 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26530e4206b34be29c7e822aa4ed1c93 consensus state has no leader
I20250625 13:52:14.165037 2348 raft_consensus.cc:513] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.166003 2348 leader_election.cc:290] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.163797 2254 rpcz_store.cc:276] 0625 13:52:12.533251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.533487 (+ 236us) service_pool.cc:225] Handling call
0625 13:52:14.163498 (+1630011us) ts_tablet_manager.cc:1145] Deleting tablet e75bb35262df49c2bd68ad00b19733fd
0625 13:52:14.163538 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.163667 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1629793}
I20250625 13:52:14.166690 882 tablet_service.cc:1515] Processing DeleteTablet for tablet d7f68a0c4bb341a9a90cc2be0f78daca with delete_type TABLET_DATA_DELETED (replaced by tablet 0c49eab755dd4f1cb16da4e10a24eecb) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:14.167054 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e75bb35262df49c2bd68ad00b19733fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e75bb35262df49c2bd68ad00b19733fd already in progress: creating tablet
W20250625 13:52:14.167191 985 leader_election.cc:343] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 38a23b08a03b48eaa051f670eb7dac65
I20250625 13:52:14.167496 2210 raft_consensus.cc:491] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.167690 2210 raft_consensus.cc:513] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.168382 2210 leader_election.cc:290] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.168665 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac4d368347444846880954ced9651404" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:14.168908 915 raft_consensus.cc:2466] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:14.169230 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2404e712e18146b68757bd77362c1358" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:14.169458 913 raft_consensus.cc:2466] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:14.169759 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6afb92085aaa47bd961481fae3d17266"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.170356 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6afb92085aaa47bd961481fae3d17266 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.171960 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048] alter complete (version 1)
I20250625 13:52:14.172202 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1096) took 1512 ms. Trace:
I20250625 13:52:14.172386 2254 rpcz_store.cc:276] 0625 13:52:12.655725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.655778 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:14.167432 (+1511654us) ts_tablet_manager.cc:1145] Deleting tablet a85fa343e30e4e838ab81cd775c298ba
0625 13:52:14.167444 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.167908 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510353}
W20250625 13:52:14.173089 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a85fa343e30e4e838ab81cd775c298ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a85fa343e30e4e838ab81cd775c298ba already in progress: creating tablet
I20250625 13:52:14.174551 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_41a10889a8c045adb2961b8b2cde45c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.175165 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41a10889a8c045adb2961b8b2cde45c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.175864 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8d38330d3fb546dfb11c9ef7f6359da0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.176285 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d38330d3fb546dfb11c9ef7f6359da0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.177570 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet fc9a79f066e84a7b87439ea4516f0287
I20250625 13:52:14.177482 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1095) took 1521 ms. Trace:
I20250625 13:52:14.178563 1171 tablet_bootstrap.cc:492] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.178715 756 tablet_service.cc:1515] Processing DeleteTablet for tablet fc9a79f066e84a7b87439ea4516f0287 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.178714 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet fc9a79f066e84a7b87439ea4516f0287 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.178835 1171 ts_tablet_manager.cc:1397] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.206s user 0.002s sys 0.003s
I20250625 13:52:14.179562 1171 raft_consensus.cc:357] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.179805 1171 raft_consensus.cc:383] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.177910 2254 rpcz_store.cc:276] 0625 13:52:12.655591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.657464 (+ 1873us) service_pool.cc:225] Handling call
0625 13:52:14.177238 (+1519774us) ts_tablet_manager.cc:1145] Deleting tablet e597f8cd8e5a4ec6be8db57f65c53c65
0625 13:52:14.177248 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.177434 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519581}
I20250625 13:52:14.179914 1171 raft_consensus.cc:738] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.180112 1171 consensus_queue.cc:260] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:14.180398 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e597f8cd8e5a4ec6be8db57f65c53c65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e597f8cd8e5a4ec6be8db57f65c53c65 already in progress: creating tablet
I20250625 13:52:14.180742 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 614181c6e4d24e7b9e5da3b4818cf065 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:14.180758 1171 ts_tablet_manager.cc:1428] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:14.181044 1171 tablet_bootstrap.cc:492] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.181315 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f7f62312f5ed44fa821485933ffc5d0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.181746 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7f62312f5ed44fa821485933ffc5d0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.182356 1171 tablet_bootstrap.cc:654] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.182271 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1094) took 1535 ms. Trace:
I20250625 13:52:14.182899 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4abaf88f43b74ca7b2cdc17a89443840" } new_table_name: "default.Table_a85c5aeb7cd742e1ad82b20001c5c2cc" modify_external_catalogs: true
I20250625 13:52:14.184286 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cf8d3ee817784b2aa014fc65c502b75c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.184681 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf8d3ee817784b2aa014fc65c502b75c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.183104 2254 rpcz_store.cc:276] 0625 13:52:12.646857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.651358 (+ 4501us) service_pool.cc:225] Handling call
0625 13:52:14.181472 (+1530114us) ts_tablet_manager.cc:1145] Deleting tablet 6340de862fd04c12bb7f85557cfe53d2
0625 13:52:14.181484 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.182248 (+ 764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519725}
I20250625 13:52:14.189765 882 tablet_service.cc:1515] Processing DeleteTablet for tablet fc9a79f066e84a7b87439ea4516f0287 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:14.189849 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6340de862fd04c12bb7f85557cfe53d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6340de862fd04c12bb7f85557cfe53d2 already in progress: creating tablet
I20250625 13:52:14.191912 2254 tablet_replica.cc:331] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:14.193858 2254 raft_consensus.cc:2241] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.194237 1165 ts_tablet_manager.cc:1918] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:14.196262 1165 log.cc:1199] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/6d95a376304e464ab75e2e1e00e7a924
I20250625 13:52:14.196136 2254 raft_consensus.cc:2270] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.200026 656 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet fc9a79f066e84a7b87439ea4516f0287
W20250625 13:52:14.200202 851 leader_election.cc:343] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:14.200697 756 tablet_service.cc:1515] Processing DeleteTablet for tablet fc9a79f066e84a7b87439ea4516f0287 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.200320 851 leader_election.cc:304] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.201710 2204 raft_consensus.cc:2747] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.201974 882 tablet_service.cc:1515] Processing DeleteTablet for tablet fc9a79f066e84a7b87439ea4516f0287 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.202145 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet fc9a79f066e84a7b87439ea4516f0287 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.202361 756 tablet_service.cc:1468] Processing CreateTablet for tablet 27be35281ffb41f98b9643828a646dbb (DEFAULT_TABLE table=default.Table_f1c1affc7c6a41e5b6d3bcb967d503da [id=c712dac2eeea40eab9e535e6bc21f775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.203200 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 27be35281ffb41f98b9643828a646dbb (DEFAULT_TABLE table=default.Table_f1c1affc7c6a41e5b6d3bcb967d503da [id=c712dac2eeea40eab9e535e6bc21f775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.203613 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27be35281ffb41f98b9643828a646dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.205889 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_18f5eaac4ead4aa8bb9e663f4fec5d79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.206452 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18f5eaac4ead4aa8bb9e663f4fec5d79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.209293 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 8063122ea7b24b6790f5c3cc884e09be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:14.209775 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5592ab667b04932896addf7514af24e consensus state has no leader
I20250625 13:52:14.210170 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7c2b5fe5db414c4c9cd1789bdf9f9822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.210537 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac4d368347444846880954ced9651404" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:14.210703 1044 raft_consensus.cc:2466] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:14.210717 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c2b5fe5db414c4c9cd1789bdf9f9822 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.211519 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27be35281ffb41f98b9643828a646dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.212488 1165 ts_tablet_manager.cc:1939] T 6d95a376304e464ab75e2e1e00e7a924 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:14.213271 882 tablet_service.cc:1468] Processing CreateTablet for tablet 27be35281ffb41f98b9643828a646dbb (DEFAULT_TABLE table=default.Table_f1c1affc7c6a41e5b6d3bcb967d503da [id=c712dac2eeea40eab9e535e6bc21f775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.213675 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27be35281ffb41f98b9643828a646dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.214721 1166 tablet_bootstrap.cc:654] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.215596 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3572db9210914a7f95e4f50317b9f680 (DEFAULT_TABLE table=default.Table_520b3c352ef6433db536a0eca3be9231 [id=c4fbc2833ef9432a90fce3767af27216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.215962 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3572db9210914a7f95e4f50317b9f680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.219714 1168 ts_tablet_manager.cc:1918] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:14.219815 1168 log.cc:1199] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/be3313ec5dfb41b68662d3ff4c8c94f9
I20250625 13:52:14.220324 1168 ts_tablet_manager.cc:1939] T be3313ec5dfb41b68662d3ff4c8c94f9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:14.220507 894 tablet_service.cc:1468] Processing CreateTablet for tablet 3572db9210914a7f95e4f50317b9f680 (DEFAULT_TABLE table=default.Table_520b3c352ef6433db536a0eca3be9231 [id=c4fbc2833ef9432a90fce3767af27216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.220880 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3572db9210914a7f95e4f50317b9f680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.221239 2338 raft_consensus.cc:491] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.221349 2338 raft_consensus.cc:513] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.221846 2338 leader_election.cc:290] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.222379 849 leader_election.cc:304] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:14.222563 2338 raft_consensus.cc:2802] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:14.222616 2338 raft_consensus.cc:491] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:14.222664 2338 raft_consensus.cc:3058] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.225298 2338 raft_consensus.cc:513] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.225819 2338 leader_election.cc:290] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.228704 2338 consensus_queue.cc:1035] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:14.229959 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 5e6886d351a24357b2d292925a2e7681 (DEFAULT_TABLE table=default.Table_42544de6446c4aed9bddadf23288264f [id=1239f72862034c4a8c5e12b2c8f112b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.230355 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e6886d351a24357b2d292925a2e7681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.230842 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_64ba301a940e4256a5cc699fbbcbaef5" } new_table_name: "default.Table_635e90a4272746cab8e8f9802215e784" modify_external_catalogs: true
I20250625 13:52:14.232157 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_03fee4b89d0c4186beb0de611b2fc05f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.232659 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03fee4b89d0c4186beb0de611b2fc05f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.232859 983 leader_election.cc:304] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.233260 983 leader_election.cc:304] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.233481 2348 raft_consensus.cc:2802] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:14.233548 2348 raft_consensus.cc:491] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:14.233616 2300 raft_consensus.cc:2802] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:14.233794 2300 raft_consensus.cc:491] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:14.233847 2300 raft_consensus.cc:3058] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.235920 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e7a56419acce4220aed05859bbdad3bd" } new_table_name: "default.Table_a6aac8f457be47e287a698c4c4df8582" modify_external_catalogs: true
I20250625 13:52:14.236268 2300 raft_consensus.cc:513] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.237111 2300 leader_election.cc:290] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.237430 1165 tablet_replica.cc:331] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:14.237635 1165 raft_consensus.cc:2241] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.237950 1165 raft_consensus.cc:2270] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.239339 1165 ts_tablet_manager.cc:1905] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:14.239750 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]) successfully deleted
W20250625 13:52:14.240038 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 55a254d82f764aa2860e465b4857a413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55a254d82f764aa2860e465b4857a413 already in progress: creating tablet
I20250625 13:52:14.240036 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5e6886d351a24357b2d292925a2e7681 (DEFAULT_TABLE table=default.Table_42544de6446c4aed9bddadf23288264f [id=1239f72862034c4a8c5e12b2c8f112b3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.240252 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6c331c57bff44091be2849c21d6ab2d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c331c57bff44091be2849c21d6ab2d7 already in progress: creating tablet
I20250625 13:52:14.240383 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e6886d351a24357b2d292925a2e7681. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:14.240459 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6fcc17f839504d6b85526b638763ec5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fcc17f839504d6b85526b638763ec5b already in progress: creating tablet
W20250625 13:52:14.240691 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d35517bbf1cb4695ae46094dc883175f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35517bbf1cb4695ae46094dc883175f already in progress: creating tablet
W20250625 13:52:14.240906 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a7b72345e04445f7a393c469a348d126 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7b72345e04445f7a393c469a348d126 already in progress: creating tablet
I20250625 13:52:14.233613 2348 raft_consensus.cc:3058] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.246037 2254 ts_tablet_manager.cc:1905] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:14.246098 2348 raft_consensus.cc:513] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.247123 2348 leader_election.cc:290] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.248771 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 875) took 4610 ms. Trace:
I20250625 13:52:14.249164 753 tablet_service.cc:1468] Processing CreateTablet for tablet 3572db9210914a7f95e4f50317b9f680 (DEFAULT_TABLE table=default.Table_520b3c352ef6433db536a0eca3be9231 [id=c4fbc2833ef9432a90fce3767af27216]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.250252 985 leader_election.cc:343] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ac4d368347444846880954ced9651404
I20250625 13:52:14.252063 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3572db9210914a7f95e4f50317b9f680. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:14.252604 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df72a1a461aa44fe80cff6e3870da8e7 consensus state has no leader
I20250625 13:52:14.253016 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 6de4b63b3e20410e8b146cf15d97d825 (DEFAULT_TABLE table=default.Table_06ce6be66e684fce8aa585310d55aba2 [id=13a32c19e3f84808a5a3fa2f192b7984]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.253645 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de4b63b3e20410e8b146cf15d97d825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.253638 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5e6886d351a24357b2d292925a2e7681 (DEFAULT_TABLE table=default.Table_42544de6446c4aed9bddadf23288264f [id=1239f72862034c4a8c5e12b2c8f112b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.248867 1168 rpcz_store.cc:276] 0625 13:52:09.637931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.637991 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:13.818311 (+4180320us) ts_tablet_manager.cc:1145] Deleting tablet be3313ec5dfb41b68662d3ff4c8c94f9
0625 13:52:13.818320 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.056119 (+237799us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.219625 (+163506us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.248748 (+ 29123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4180129}
I20250625 13:52:14.254544 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e6886d351a24357b2d292925a2e7681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.254673 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet be3313ec5dfb41b68662d3ff4c8c94f9 (table default.Table_fb311c42df024421b325596e0cb7a778 [id=7b67603755e7428496524ee716b435e9]) successfully deleted
I20250625 13:52:14.255038 1168 tablet_replica.cc:331] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:14.255446 1168 raft_consensus.cc:2241] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.255844 1168 raft_consensus.cc:2270] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.256549 1168 ts_tablet_manager.cc:1905] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:14.260264 1166 tablet_bootstrap.cc:492] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:14.260422 1166 ts_tablet_manager.cc:1397] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.004s
I20250625 13:52:14.261257 1166 raft_consensus.cc:357] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.261451 1166 raft_consensus.cc:383] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.261528 1166 raft_consensus.cc:738] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.261754 1166 consensus_queue.cc:260] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.262382 1166 ts_tablet_manager.cc:1428] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:14.262653 1166 tablet_bootstrap.cc:492] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:14.264700 1165 ts_tablet_manager.cc:1918] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.264806 1165 log.cc:1199] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/da2f8d0d9a714a28ba6fb9642d461de2
I20250625 13:52:14.264921 1166 tablet_bootstrap.cc:654] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.265108 1165 ts_tablet_manager.cc:1939] T da2f8d0d9a714a28ba6fb9642d461de2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:14.265491 1165 tablet_replica.cc:331] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:14.265672 1165 raft_consensus.cc:2241] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.265744 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet da2f8d0d9a714a28ba6fb9642d461de2 (table default.Table_dc86aa488ba944c689ae858bc3800718 [id=028f5254a2c045929bdb41c0b273875c]) successfully deleted
I20250625 13:52:14.265836 1165 raft_consensus.cc:2270] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.265779 753 tablet_service.cc:1468] Processing CreateTablet for tablet 6de4b63b3e20410e8b146cf15d97d825 (DEFAULT_TABLE table=default.Table_06ce6be66e684fce8aa585310d55aba2 [id=13a32c19e3f84808a5a3fa2f192b7984]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.266145 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de4b63b3e20410e8b146cf15d97d825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.266222 894 tablet_service.cc:1468] Processing CreateTablet for tablet 6de4b63b3e20410e8b146cf15d97d825 (DEFAULT_TABLE table=default.Table_06ce6be66e684fce8aa585310d55aba2 [id=13a32c19e3f84808a5a3fa2f192b7984]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.266585 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6de4b63b3e20410e8b146cf15d97d825. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.269721 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bf56f0c20508449b8c381ba2c3510578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.270285 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf56f0c20508449b8c381ba2c3510578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.273104 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 62822a0a8bf54f0184312ddddfb561aa (DEFAULT_TABLE table=default.Table_e85e50c4b7e144caa0afdfc419814cae [id=d764eba57661411a965da7ec8487d5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.273566 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62822a0a8bf54f0184312ddddfb561aa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:14.273593 985 leader_election.cc:343] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2404e712e18146b68757bd77362c1358
W20250625 13:52:14.274909 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 52541.2us
I20250625 13:52:14.279759 1165 ts_tablet_manager.cc:1905] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:14.283579 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0a13c34b24244d58daa0d67589b5100 consensus state has no leader
I20250625 13:52:14.284736 886 tablet_service.cc:1468] Processing CreateTablet for tablet 62822a0a8bf54f0184312ddddfb561aa (DEFAULT_TABLE table=default.Table_e85e50c4b7e144caa0afdfc419814cae [id=d764eba57661411a965da7ec8487d5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.285157 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62822a0a8bf54f0184312ddddfb561aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.287161 1046 raft_consensus.cc:3058] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.292062 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68b8c8aa8036454298bf1e81bab91c01" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:14.294113 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac4d368347444846880954ced9651404" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:14.294399 1044 raft_consensus.cc:2391] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate affa1c3035b940e88ca143be7d35840d in current term 1: Already voted for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in this term.
I20250625 13:52:14.293190 753 tablet_service.cc:1468] Processing CreateTablet for tablet 62822a0a8bf54f0184312ddddfb561aa (DEFAULT_TABLE table=default.Table_e85e50c4b7e144caa0afdfc419814cae [id=d764eba57661411a965da7ec8487d5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.302778 1168 ts_tablet_manager.cc:1918] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.302822 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
I20250625 13:52:14.304432 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62822a0a8bf54f0184312ddddfb561aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.311972 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 9a2007e51d424053bc2836dedf46add2 (table default.Table_4a226ca5c5c84ef380d6c37dd08f6740 [id=fa06a5b0571b4a3a8b9b68f1f05fc3e6]). Expected version 1 got 0
I20250625 13:52:14.313248 1170 tablet_bootstrap.cc:492] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:14.313398 1170 ts_tablet_manager.cc:1397] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.262s user 0.010s sys 0.002s
I20250625 13:52:14.314461 744 tablet_service.cc:1468] Processing CreateTablet for tablet 3db88a21fc9f43cbb524356c06827618 (DEFAULT_TABLE table=default.Table_154bff2914d341e5b12048cfbb81b179 [id=b1f30b9b952f42ceb70bfe4a61379d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.314853 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db88a21fc9f43cbb524356c06827618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.316140 751 tablet_service.cc:1468] Processing CreateTablet for tablet 1d34a4de076842a5966df1dc28548a4c (DEFAULT_TABLE table=default.Table_262ec080b91d44d68afd7d9c6341cb25 [id=29c869a5bbe04125ac60195cb120d423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.317086 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d34a4de076842a5966df1dc28548a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.318233 757 tablet_service.cc:1468] Processing CreateTablet for tablet 7e4fbede3f9040709ae4bdc9bbb792ad (DEFAULT_TABLE table=default.Table_6bd08e23382b4c0880724d8266377783 [id=41046c948fdf454f93b5bfcae3af5990]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.318538 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4fbede3f9040709ae4bdc9bbb792ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.319581 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 1d34a4de076842a5966df1dc28548a4c (DEFAULT_TABLE table=default.Table_262ec080b91d44d68afd7d9c6341cb25 [id=29c869a5bbe04125ac60195cb120d423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.319923 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d34a4de076842a5966df1dc28548a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.322495 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7e4fbede3f9040709ae4bdc9bbb792ad (DEFAULT_TABLE table=default.Table_6bd08e23382b4c0880724d8266377783 [id=41046c948fdf454f93b5bfcae3af5990]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.322860 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4fbede3f9040709ae4bdc9bbb792ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.331692 1170 raft_consensus.cc:357] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.331956 1170 raft_consensus.cc:383] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.332047 1170 raft_consensus.cc:738] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.332296 1170 consensus_queue.cc:260] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.333328 1170 ts_tablet_manager.cc:1428] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250625 13:52:14.333621 1170 tablet_bootstrap.cc:492] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:14.334015 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_66dd8f78c67643d9828dee272954c648" } modify_external_catalogs: true
I20250625 13:52:14.334420 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_66dd8f78c67643d9828dee272954c648" } modify_external_catalogs: true
I20250625 13:52:14.336661 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d8054fa43c994aab98b1d5541d0be6f9" } modify_external_catalogs: true
I20250625 13:52:14.336825 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d8054fa43c994aab98b1d5541d0be6f9" } modify_external_catalogs: true
I20250625 13:52:14.341513 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f26987562c25452388d457eb3b600595" } modify_external_catalogs: true
I20250625 13:52:14.341684 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f26987562c25452388d457eb3b600595" } modify_external_catalogs: true
I20250625 13:52:14.344166 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3562e8b8731a4d76bb1a36189f957e07
I20250625 13:52:14.345228 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 3562e8b8731a4d76bb1a36189f957e07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.345577 1168 log.cc:1199] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:14.345947 1168 ts_tablet_manager.cc:1939] T 042359eacaa2451abc0fa6ed447a2905 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:14.346693 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4d11b567f2c14b62a446981c6ad894d4" } modify_external_catalogs: true
I20250625 13:52:14.346853 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4d11b567f2c14b62a446981c6ad894d4" } modify_external_catalogs: true
I20250625 13:52:14.348178 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8f88091205b944b386094e50cf92c36b
I20250625 13:52:14.349124 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f88091205b944b386094e50cf92c36b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.349395 1166 tablet_bootstrap.cc:492] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:14.349538 1166 ts_tablet_manager.cc:1397] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.000s
I20250625 13:52:14.350349 1166 raft_consensus.cc:357] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.350515 1166 raft_consensus.cc:383] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.350576 1166 raft_consensus.cc:738] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.350576 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 39ac79102eab46688e767faed6d6f114
I20250625 13:52:14.350782 1166 consensus_queue.cc:260] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.314879 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3db88a21fc9f43cbb524356c06827618 (DEFAULT_TABLE table=default.Table_154bff2914d341e5b12048cfbb81b179 [id=b1f30b9b952f42ceb70bfe4a61379d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.351434 1166 ts_tablet_manager.cc:1428] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:14.351619 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 529c3bf8e72f46948cfb864445ff51f3 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:14.351658 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db88a21fc9f43cbb524356c06827618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.351694 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 39ac79102eab46688e767faed6d6f114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.352530 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 39ac79102eab46688e767faed6d6f114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.352836 1166 tablet_bootstrap.cc:492] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:14.354077 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3db88a21fc9f43cbb524356c06827618 (DEFAULT_TABLE table=default.Table_154bff2914d341e5b12048cfbb81b179 [id=b1f30b9b952f42ceb70bfe4a61379d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.354777 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db88a21fc9f43cbb524356c06827618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.354902 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 39ac79102eab46688e767faed6d6f114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.356593 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d0174345facc4216b6ddeb8f8c73790d
I20250625 13:52:14.357699 879 tablet_service.cc:1515] Processing DeleteTablet for tablet d0174345facc4216b6ddeb8f8c73790d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.358291 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d0174345facc4216b6ddeb8f8c73790d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.359143 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 906) took 4537 ms. Trace:
I20250625 13:52:14.359251 1168 rpcz_store.cc:276] 0625 13:52:09.821839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.821888 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:14.254991 (+4433103us) ts_tablet_manager.cc:1145] Deleting tablet 042359eacaa2451abc0fa6ed447a2905
0625 13:52:14.255002 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.299518 (+ 44516us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.302686 (+ 3168us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.359126 (+ 56440us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4432832}
I20250625 13:52:14.359760 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 042359eacaa2451abc0fa6ed447a2905 (table default.Table_3c1061dac55a4e97ad139950a7c8f302 [id=426662edb72f4d1aa3e0aa1ee934e62d]) successfully deleted
I20250625 13:52:14.360428 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 68e31d35a07843e689c5244796b567d8 (DEFAULT_TABLE table=default.Table_8a3bb463510942988dfd2cb114c96e48 [id=359a935f26f8424e94c368599b410622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.360831 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e31d35a07843e689c5244796b567d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.360909 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d0174345facc4216b6ddeb8f8c73790d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.315172 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2404e712e18146b68757bd77362c1358" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:14.315474 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac4d368347444846880954ced9651404" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:14.361539 757 tablet_service.cc:1468] Processing CreateTablet for tablet fc086d416d0f475987bd67f122487353 (DEFAULT_TABLE table=default.Table_d070a8c36af544c6b3795695527477aa [id=7c71ea2ef29a49e7bc1ba091a715d130]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.361732 914 raft_consensus.cc:2391] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:52:14.361850 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc086d416d0f475987bd67f122487353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.362095 915 raft_consensus.cc:3058] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.362440 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9c571f5508ab481b93e71babb1f65e78 (DEFAULT_TABLE table=default.Table_0a691ba3cecb4068b1955aa46a0ead68 [id=15bd5db613c34526be4805b87b190faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.362725 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c571f5508ab481b93e71babb1f65e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.364382 896 tablet_service.cc:1468] Processing CreateTablet for tablet fc086d416d0f475987bd67f122487353 (DEFAULT_TABLE table=default.Table_d070a8c36af544c6b3795695527477aa [id=7c71ea2ef29a49e7bc1ba091a715d130]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.364701 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc086d416d0f475987bd67f122487353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.366384 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e85e50c4b7e144caa0afdfc419814cae" } new_table_name: "default.Table_46ba401738b64322afb12c3635116d5c" modify_external_catalogs: true
I20250625 13:52:14.368214 879 tablet_service.cc:1468] Processing CreateTablet for tablet 68e31d35a07843e689c5244796b567d8 (DEFAULT_TABLE table=default.Table_8a3bb463510942988dfd2cb114c96e48 [id=359a935f26f8424e94c368599b410622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.368564 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e31d35a07843e689c5244796b567d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.369390 915 raft_consensus.cc:2466] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:14.316815 851 leader_election.cc:343] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ac4d368347444846880954ced9651404
I20250625 13:52:14.316936 890 tablet_service.cc:1468] Processing CreateTablet for tablet 1d34a4de076842a5966df1dc28548a4c (DEFAULT_TABLE table=default.Table_262ec080b91d44d68afd7d9c6341cb25 [id=29c869a5bbe04125ac60195cb120d423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.371093 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d34a4de076842a5966df1dc28548a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.371927 744 tablet_service.cc:1468] Processing CreateTablet for tablet 68e31d35a07843e689c5244796b567d8 (DEFAULT_TABLE table=default.Table_8a3bb463510942988dfd2cb114c96e48 [id=359a935f26f8424e94c368599b410622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.372274 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e31d35a07843e689c5244796b567d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.374920 1165 ts_tablet_manager.cc:1918] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.375011 1165 log.cc:1199] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/b7d888e628a94f939ecf2571e60b1948
I20250625 13:52:14.375283 1165 ts_tablet_manager.cc:1939] T b7d888e628a94f939ecf2571e60b1948 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:14.376160 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet b7d888e628a94f939ecf2571e60b1948 (table default.Table_7a2ca389ecd148b9be0e90c130fe6431 [id=26d0bdba8be543d89d5337c86bbf6e57]) successfully deleted
W20250625 13:52:14.376566 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 19f1050a29b540558aaf1e31118e4531 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f1050a29b540558aaf1e31118e4531 already in progress: creating tablet
W20250625 13:52:14.376818 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1f47260bf0ad41fdbfbad50380a12547 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f47260bf0ad41fdbfbad50380a12547 already in progress: creating tablet
W20250625 13:52:14.377043 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2b29392cfeb14f46a60fd0731f220127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b29392cfeb14f46a60fd0731f220127 already in progress: creating tablet
I20250625 13:52:14.377049 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_83060ba51717429e8656da842eac7d78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.377250 1018 tablet_service.cc:1468] Processing CreateTablet for tablet fc086d416d0f475987bd67f122487353 (DEFAULT_TABLE table=default.Table_d070a8c36af544c6b3795695527477aa [id=7c71ea2ef29a49e7bc1ba091a715d130]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.377547 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc already in progress: creating tablet
I20250625 13:52:14.377630 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc086d416d0f475987bd67f122487353. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:14.377631 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83060ba51717429e8656da842eac7d78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.379376 2338 raft_consensus.cc:491] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.379506 2338 raft_consensus.cc:513] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.380071 2338 leader_election.cc:290] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:14.381915 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62822a0a8bf54f0184312ddddfb561aa consensus state has no leader
I20250625 13:52:14.383499 1165 tablet_replica.cc:331] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:14.383693 1165 raft_consensus.cc:2241] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.383836 1165 raft_consensus.cc:2270] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.384080 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 3562e8b8731a4d76bb1a36189f957e07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.385068 2254 ts_tablet_manager.cc:1918] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.385174 2254 log.cc:1199] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a0e81d4c904248eba858462bcf31c5d9
I20250625 13:52:14.385439 2254 ts_tablet_manager.cc:1939] T a0e81d4c904248eba858462bcf31c5d9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:14.385736 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1104) took 1703 ms. Trace:
I20250625 13:52:14.386793 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:14.387480 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_612bc80f50694afebddc36a5f827e377"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.387544 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_92b3804731fc429a8b6412d0be16b8ad" } new_table_name: "default.Table_c0213ee08c5b4b58bc22aec0259011e9" modify_external_catalogs: true
W20250625 13:52:14.388029 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_612bc80f50694afebddc36a5f827e377 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:14.388327 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 71510.1us
I20250625 13:52:14.388644 2338 consensus_queue.cc:1035] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:14.326392 882 tablet_service.cc:1468] Processing CreateTablet for tablet 7e4fbede3f9040709ae4bdc9bbb792ad (DEFAULT_TABLE table=default.Table_6bd08e23382b4c0880724d8266377783 [id=41046c948fdf454f93b5bfcae3af5990]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.345425 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 3562e8b8731a4d76bb1a36189f957e07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.390328 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4fbede3f9040709ae4bdc9bbb792ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.392071 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2c311e79aef2487d8524720a4cd0677c" } modify_external_catalogs: true
I20250625 13:52:14.392319 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2c311e79aef2487d8524720a4cd0677c" } modify_external_catalogs: true
I20250625 13:52:14.397553 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d43ea6e00a7d40baa9e849a59f0bd42b (DEFAULT_TABLE table=default.Table_d1c8466871174daea32ea7d8bac86daa [id=25ae672e5ac043f2a1b5ca08437d325b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.397964 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43ea6e00a7d40baa9e849a59f0bd42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.398130 894 tablet_service.cc:1468] Processing CreateTablet for tablet d43ea6e00a7d40baa9e849a59f0bd42b (DEFAULT_TABLE table=default.Table_d1c8466871174daea32ea7d8bac86daa [id=25ae672e5ac043f2a1b5ca08437d325b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.398154 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1c8c5b5bb73e49839566884431e4b9d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.398458 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43ea6e00a7d40baa9e849a59f0bd42b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:14.398661 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c8c5b5bb73e49839566884431e4b9d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.399103 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 3abfa19eef7540b69f1bec814a1714ae (DEFAULT_TABLE table=default.Table_3c06b182c7854a5380dcb729131218d1 [id=8e912c71fb614c29ac928cee6c2efb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.399457 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abfa19eef7540b69f1bec814a1714ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.400255 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 9c571f5508ab481b93e71babb1f65e78 (DEFAULT_TABLE table=default.Table_0a691ba3cecb4068b1955aa46a0ead68 [id=15bd5db613c34526be4805b87b190faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.400597 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c571f5508ab481b93e71babb1f65e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.402431 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 758d2e5d32e7427192bd9b7fd18a9382 (DEFAULT_TABLE table=default.Table_ec05413cb661420684670fce5e9b6107 [id=6e3b37e354e2430d94238c45278d5ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.402756 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758d2e5d32e7427192bd9b7fd18a9382. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.403609 757 tablet_service.cc:1468] Processing CreateTablet for tablet 758d2e5d32e7427192bd9b7fd18a9382 (DEFAULT_TABLE table=default.Table_ec05413cb661420684670fce5e9b6107 [id=6e3b37e354e2430d94238c45278d5ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.403702 896 tablet_service.cc:1468] Processing CreateTablet for tablet 3abfa19eef7540b69f1bec814a1714ae (DEFAULT_TABLE table=default.Table_3c06b182c7854a5380dcb729131218d1 [id=8e912c71fb614c29ac928cee6c2efb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.404026 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abfa19eef7540b69f1bec814a1714ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.405043 758 tablet_service.cc:1468] Processing CreateTablet for tablet eeda22a8a9d147919fd110c2d83af5ae (DEFAULT_TABLE table=default.Table_60d3522fb58e4d2d8478eec6df099e91 [id=2f7c46bb4abc45d4b800b144cdd4874f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.385819 2254 rpcz_store.cc:276] 0625 13:52:12.682431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.682500 (+ 69us) service_pool.cc:225] Handling call
0625 13:52:14.190971 (+1508471us) ts_tablet_manager.cc:1145] Deleting tablet a0e81d4c904248eba858462bcf31c5d9
0625 13:52:14.190982 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.281052 (+ 90070us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.384989 (+103937us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.385718 (+ 729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507924}
I20250625 13:52:14.405839 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a0e81d4c904248eba858462bcf31c5d9 (table default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]) successfully deleted
I20250625 13:52:14.406315 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1099) took 1740 ms. Trace:
I20250625 13:52:14.406384 2254 rpcz_store.cc:276] 0625 13:52:12.665660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.685175 (+ 19515us) service_pool.cc:225] Handling call
0625 13:52:14.406197 (+1721022us) ts_tablet_manager.cc:1145] Deleting tablet ef1f9c03b0ef455cad05b3fe59bdd67c
0625 13:52:14.406206 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.406299 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1720381}
W20250625 13:52:14.406805 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet ef1f9c03b0ef455cad05b3fe59bdd67c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef1f9c03b0ef455cad05b3fe59bdd67c already in progress: creating tablet
I20250625 13:52:14.407131 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1097) took 1750 ms. Trace:
I20250625 13:52:14.407202 2254 rpcz_store.cc:276] 0625 13:52:12.657005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.685857 (+ 28852us) service_pool.cc:225] Handling call
0625 13:52:14.407031 (+1721174us) ts_tablet_manager.cc:1145] Deleting tablet f3c75deb64c74f0282efeed827659ec2
0625 13:52:14.407040 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.407116 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1719424}
W20250625 13:52:14.407615 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f3c75deb64c74f0282efeed827659ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3c75deb64c74f0282efeed827659ec2 already in progress: creating tablet
I20250625 13:52:14.407922 2348 raft_consensus.cc:491] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.408056 2348 raft_consensus.cc:513] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.409108 2348 leader_election.cc:290] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:14.314642 849 leader_election.cc:343] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:14.410792 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 96152.4us
W20250625 13:52:14.411094 849 consensus_peers.cc:487] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:14.411314 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a2251328ba8745ae98b9fead6e8e516e" } new_table_name: "default.Table_e43ad6762bd943779162fd6eddfe14f4" modify_external_catalogs: true
I20250625 13:52:14.412236 879 tablet_service.cc:1468] Processing CreateTablet for tablet 9c571f5508ab481b93e71babb1f65e78 (DEFAULT_TABLE table=default.Table_0a691ba3cecb4068b1955aa46a0ead68 [id=15bd5db613c34526be4805b87b190faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.412627 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c571f5508ab481b93e71babb1f65e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.412633 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "430c3bed360b4f32855a8adb99076608" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:14.412940 751 tablet_service.cc:1468] Processing CreateTablet for tablet 3abfa19eef7540b69f1bec814a1714ae (DEFAULT_TABLE table=default.Table_3c06b182c7854a5380dcb729131218d1 [id=8e912c71fb614c29ac928cee6c2efb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.314854 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38a23b08a03b48eaa051f670eb7dac65" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:14.349329 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f88091205b944b386094e50cf92c36b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.413758 913 raft_consensus.cc:3058] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.415565 753 tablet_service.cc:1468] Processing CreateTablet for tablet d43ea6e00a7d40baa9e849a59f0bd42b (DEFAULT_TABLE table=default.Table_d1c8466871174daea32ea7d8bac86daa [id=25ae672e5ac043f2a1b5ca08437d325b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.416008 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 908) took 4335 ms. Trace:
I20250625 13:52:14.416095 1168 rpcz_store.cc:276] 0625 13:52:10.081009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.081076 (+ 67us) service_pool.cc:225] Handling call
0625 13:52:14.415852 (+4334776us) ts_tablet_manager.cc:1145] Deleting tablet 753e4767d7b84515ad2c93437e46c65b
0625 13:52:14.415863 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.415989 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4334413}
I20250625 13:52:14.416339 913 raft_consensus.cc:2466] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:14.416503 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 753e4767d7b84515ad2c93437e46c65b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753e4767d7b84515ad2c93437e46c65b already in progress: creating tablet
I20250625 13:52:14.416611 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 909) took 4333 ms. Trace:
I20250625 13:52:14.416690 1168 rpcz_store.cc:276] 0625 13:52:10.083246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.083325 (+ 79us) service_pool.cc:225] Handling call
0625 13:52:14.416484 (+4333159us) ts_tablet_manager.cc:1145] Deleting tablet a89f9ffb5e4643f8915e9fd69dd3e842
0625 13:52:14.416493 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.416594 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4332813}
W20250625 13:52:14.416968 985 leader_election.cc:343] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d43ba17ebfec4244894d1b6e23200c33
W20250625 13:52:14.417024 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a89f9ffb5e4643f8915e9fd69dd3e842 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a89f9ffb5e4643f8915e9fd69dd3e842 already in progress: creating tablet
I20250625 13:52:14.417099 1168 tablet_replica.cc:331] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:14.417279 1168 raft_consensus.cc:2241] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:14.417496 1168 pending_rounds.cc:70] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:14.417559 1168 pending_rounds.cc:77] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520650713989120 op_type: NO_OP noop_request { }
I20250625 13:52:14.417672 1168 raft_consensus.cc:2887] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:14.417773 1168 raft_consensus.cc:2270] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.419059 890 tablet_service.cc:1468] Processing CreateTablet for tablet 758d2e5d32e7427192bd9b7fd18a9382 (DEFAULT_TABLE table=default.Table_ec05413cb661420684670fce5e9b6107 [id=6e3b37e354e2430d94238c45278d5ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.419422 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758d2e5d32e7427192bd9b7fd18a9382. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.420819 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_eaa8f84a59064ebc8833a040f29fc7bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.421332 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaa8f84a59064ebc8833a040f29fc7bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.423198 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeda22a8a9d147919fd110c2d83af5ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:14.426313 849 leader_election.cc:343] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:14.427373 851 leader_election.cc:343] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 68b8c8aa8036454298bf1e81bab91c01
I20250625 13:52:14.427573 851 leader_election.cc:304] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.427752 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758d2e5d32e7427192bd9b7fd18a9382. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.427943 2338 raft_consensus.cc:2747] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:14.428267 851 leader_election.cc:343] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ac4d368347444846880954ced9651404
I20250625 13:52:14.428351 851 leader_election.cc:304] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.428532 2338 raft_consensus.cc:2747] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.428604 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 8e11040f23434015a5ee5391ea5861c0 (DEFAULT_TABLE table=default.Table_aeb56ffe1b1343528965b690ef571cc7 [id=4f23bc58143640329474c296ba27e8c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.428905 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8e11040f23434015a5ee5391ea5861c0 (DEFAULT_TABLE table=default.Table_aeb56ffe1b1343528965b690ef571cc7 [id=4f23bc58143640329474c296ba27e8c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.429239 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e11040f23434015a5ee5391ea5861c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.429319 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e11040f23434015a5ee5391ea5861c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.431998 2367 tablet.cc:1722] T 79beb23eebf54fd9951ea5a33fcfb0ae P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:14.432034 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8e11040f23434015a5ee5391ea5861c0 (DEFAULT_TABLE table=default.Table_aeb56ffe1b1343528965b690ef571cc7 [id=4f23bc58143640329474c296ba27e8c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.433069 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e11040f23434015a5ee5391ea5861c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.434566 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d9bf47b9c43c456e997fd5950c43f132"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.435107 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9bf47b9c43c456e997fd5950c43f132 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.436367 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_78bde809b59a4fdc8059d75304a37e00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.436925 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78bde809b59a4fdc8059d75304a37e00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.439236 2348 raft_consensus.cc:491] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.439409 2348 raft_consensus.cc:513] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.440124 2348 leader_election.cc:290] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.440606 886 tablet_service.cc:1468] Processing CreateTablet for tablet eeda22a8a9d147919fd110c2d83af5ae (DEFAULT_TABLE table=default.Table_60d3522fb58e4d2d8478eec6df099e91 [id=2f7c46bb4abc45d4b800b144cdd4874f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.440727 985 leader_election.cc:343] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 87f5b36ddb474ed6903c14876b236ba8
I20250625 13:52:14.441006 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeda22a8a9d147919fd110c2d83af5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.447594 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 529c3bf8e72f46948cfb864445ff51f3
I20250625 13:52:14.448555 1171 tablet_bootstrap.cc:492] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.448715 1171 ts_tablet_manager.cc:1397] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.268s user 0.002s sys 0.000s
I20250625 13:52:14.449043 739 tablet_service.cc:1468] Processing CreateTablet for tablet fc9a79f066e84a7b87439ea4516f0287 (DEFAULT_TABLE table=default.Table_c5d44351d1f8484294462de887c97b18 [id=ca75aecc78244b81844e7899c42b3e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.449476 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9a79f066e84a7b87439ea4516f0287. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.449467 1171 raft_consensus.cc:357] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.449636 1171 raft_consensus.cc:383] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.449687 1171 raft_consensus.cc:738] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.450367 1023 tablet_service.cc:1468] Processing CreateTablet for tablet c945668af4b64d4ca82957feda140a67 (DEFAULT_TABLE table=default.Table_8900946fe67c42d780eabc0aff65f01c [id=600001440fde445d9939e0b7d64a2a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.450769 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c945668af4b64d4ca82957feda140a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.450866 878 tablet_service.cc:1468] Processing CreateTablet for tablet c945668af4b64d4ca82957feda140a67 (DEFAULT_TABLE table=default.Table_8900946fe67c42d780eabc0aff65f01c [id=600001440fde445d9939e0b7d64a2a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.451188 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c945668af4b64d4ca82957feda140a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.452421 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f88091205b944b386094e50cf92c36b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.452505 740 tablet_service.cc:1468] Processing CreateTablet for tablet c945668af4b64d4ca82957feda140a67 (DEFAULT_TABLE table=default.Table_8900946fe67c42d780eabc0aff65f01c [id=600001440fde445d9939e0b7d64a2a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.452824 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c945668af4b64d4ca82957feda140a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.407945 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1098) took 1742 ms. Trace:
I20250625 13:52:14.453933 2254 rpcz_store.cc:276] 0625 13:52:12.665503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.687662 (+ 22159us) service_pool.cc:225] Handling call
0625 13:52:14.407849 (+1720187us) ts_tablet_manager.cc:1145] Deleting tablet bd036ec08f774f5393067ce20bb07610
0625 13:52:14.407856 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.407930 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1720009}
W20250625 13:52:14.454406 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bd036ec08f774f5393067ce20bb07610 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd036ec08f774f5393067ce20bb07610 already in progress: creating tablet
I20250625 13:52:14.455380 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 529c3bf8e72f46948cfb864445ff51f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 09733b6641dd49c9b29633af1c466aa4) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.455457 983 leader_election.cc:304] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:14.455667 2254 tablet_replica.cc:331] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:14.455843 983 leader_election.cc:304] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.455971 596 catalog_manager.cc:6652] ReplaceTablet: tablet 529c3bf8e72f46948cfb864445ff51f3 deleted and replaced by tablet 09733b6641dd49c9b29633af1c466aa4
I20250625 13:52:14.456171 983 leader_election.cc:304] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.456315 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_dec20faad45645eab2f7db58d65b3932"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.456440 2300 raft_consensus.cc:2747] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.456595 2300 raft_consensus.cc:2802] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:14.456667 2300 raft_consensus.cc:695] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
W20250625 13:52:14.456882 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dec20faad45645eab2f7db58d65b3932 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.456892 2300 consensus_queue.cc:237] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.457818 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 529c3bf8e72f46948cfb864445ff51f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 09733b6641dd49c9b29633af1c466aa4) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.457998 2204 raft_consensus.cc:491] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.458108 2204 raft_consensus.cc:513] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.458156 1014 tablet_service.cc:1468] Processing CreateTablet for tablet eeda22a8a9d147919fd110c2d83af5ae (DEFAULT_TABLE table=default.Table_60d3522fb58e4d2d8478eec6df099e91 [id=2f7c46bb4abc45d4b800b144cdd4874f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.458480 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeda22a8a9d147919fd110c2d83af5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.458647 2204 leader_election.cc:290] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.459200 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8063122ea7b24b6790f5c3cc884e09be" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:14.459882 849 leader_election.cc:343] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.461691 1018 tablet_service.cc:1468] Processing CreateTablet for tablet fc9a79f066e84a7b87439ea4516f0287 (DEFAULT_TABLE table=default.Table_c5d44351d1f8484294462de887c97b18 [id=ca75aecc78244b81844e7899c42b3e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.462064 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9a79f066e84a7b87439ea4516f0287. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.462719 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87f5b36ddb474ed6903c14876b236ba8" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:14.462935 913 raft_consensus.cc:2466] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:14.468304 755 tablet_service.cc:1468] Processing CreateTablet for tablet 29e223675c7842b383865a270b97ce58 (DEFAULT_TABLE table=default.Table_39d5b8799785461d861d7d1ef1fba8e2 [id=50e69256b6024c6eb2b5ae551edec7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.470568 1166 tablet_bootstrap.cc:654] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.470728 878 tablet_service.cc:1468] Processing CreateTablet for tablet 29e223675c7842b383865a270b97ce58 (DEFAULT_TABLE table=default.Table_39d5b8799785461d861d7d1ef1fba8e2 [id=50e69256b6024c6eb2b5ae551edec7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.471120 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29e223675c7842b383865a270b97ce58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.471558 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 29e223675c7842b383865a270b97ce58 (DEFAULT_TABLE table=default.Table_39d5b8799785461d861d7d1ef1fba8e2 [id=50e69256b6024c6eb2b5ae551edec7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.471951 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29e223675c7842b383865a270b97ce58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.475441 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:14.477763 2348 raft_consensus.cc:2802] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:14.479727 1165 ts_tablet_manager.cc:1905] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:14.479871 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29e223675c7842b383865a270b97ce58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.481048 1171 consensus_queue.cc:260] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.481715 1171 ts_tablet_manager.cc:1428] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
W20250625 13:52:14.482431 851 leader_election.cc:343] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8063122ea7b24b6790f5c3cc884e09be
I20250625 13:52:14.482530 851 leader_election.cc:304] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.482820 2204 raft_consensus.cc:2747] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.484059 1170 tablet_bootstrap.cc:654] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.486886 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abfa19eef7540b69f1bec814a1714ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.487332 879 tablet_service.cc:1468] Processing CreateTablet for tablet fc9a79f066e84a7b87439ea4516f0287 (DEFAULT_TABLE table=default.Table_c5d44351d1f8484294462de887c97b18 [id=ca75aecc78244b81844e7899c42b3e69]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.489015 851 leader_election.cc:343] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 430c3bed360b4f32855a8adb99076608
I20250625 13:52:14.489157 851 leader_election.cc:304] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.490789 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 529c3bf8e72f46948cfb864445ff51f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 09733b6641dd49c9b29633af1c466aa4) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.491372 2204 raft_consensus.cc:2747] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.494511 1165 ts_tablet_manager.cc:1918] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:14.499650 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 43485.1us
W20250625 13:52:14.500636 983 leader_election.cc:343] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: d43ba17ebfec4244894d1b6e23200c33
I20250625 13:52:14.500877 983 leader_election.cc:304] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:14.501281 983 leader_election.cc:304] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.501637 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:14.501997 2380 raft_consensus.cc:2747] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.502193 2380 raft_consensus.cc:2802] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:14.502301 2380 raft_consensus.cc:491] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:14.502416 2380 raft_consensus.cc:3058] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.512039 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9a79f066e84a7b87439ea4516f0287. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.518129 1171 tablet_bootstrap.cc:492] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.519560 1171 tablet_bootstrap.cc:654] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.530272 2335 tablet.cc:1722] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 13:52:14.548226 1171 tablet_bootstrap.cc:492] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.548671 2271 raft_consensus.cc:491] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.548807 2271 raft_consensus.cc:513] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.549440 2271 leader_election.cc:290] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.550184 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c5fc63828d9408b9cf774a966f43cf2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:14.550944 849 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.551514 589 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 79beb23eebf54fd9951ea5a33fcfb0ae (table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]). Expected version 1 got 0
I20250625 13:52:14.551802 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
I20250625 13:52:14.552783 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_aa21254fb9d2413e9533648cba22c6aa" } new_table_name: "default.Table_a614c3633f2a46c09254c7c49ffba6e0" modify_external_catalogs: true
I20250625 13:52:14.552749 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a27119266aa84cf295b69911fabab0af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.553222 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (table default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916]). Expected version 5 got 0
W20250625 13:52:14.553371 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a27119266aa84cf295b69911fabab0af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.554231 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5027b25ff4d946e8880a10a456943c12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.554544 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5bf16bad14734a7c84f559fc18b171e7 consensus state has no leader
W20250625 13:52:14.554720 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5027b25ff4d946e8880a10a456943c12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.554946 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_80f1e7ae1dce4246b10319e14d4dcaa6" } new_table_name: "default.Table_e8bf18592c6f46869927dd4b1c328338" modify_external_catalogs: true
I20250625 13:52:14.556169 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3d63596a7f60412eafc186aec5181d38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.556934 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_34d8b1f4040c4fc08b1f847c31060596" } modify_external_catalogs: true
W20250625 13:52:14.557088 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d63596a7f60412eafc186aec5181d38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:14.557456 851 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 6c5fc63828d9408b9cf774a966f43cf2
I20250625 13:52:14.557575 851 leader_election.cc:304] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.557981 2271 raft_consensus.cc:2747] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.557456 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3f56a8a6cb964d3a979e2f1d944eeddc
W20250625 13:52:14.556561 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 754c690dd747460eafbd13ea9efdc692 consensus state has no leader
I20250625 13:52:14.557090 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_34d8b1f4040c4fc08b1f847c31060596" } modify_external_catalogs: true
I20250625 13:52:14.560592 2380 raft_consensus.cc:513] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:14.562186 985 leader_election.cc:343] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 87f5b36ddb474ed6903c14876b236ba8
I20250625 13:52:14.562527 1171 ts_tablet_manager.cc:1397] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.004s
I20250625 13:52:14.562935 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c2b5fe5db414c4c9cd1789bdf9f9822" } modify_external_catalogs: true
I20250625 13:52:14.563071 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1f9d1b5465174503915b895238d69e85" } modify_external_catalogs: true
I20250625 13:52:14.563130 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c2b5fe5db414c4c9cd1789bdf9f9822" } modify_external_catalogs: true
I20250625 13:52:14.563231 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1f9d1b5465174503915b895238d69e85" } modify_external_catalogs: true
I20250625 13:52:14.566104 1171 raft_consensus.cc:357] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.566370 1171 raft_consensus.cc:383] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.566475 1171 raft_consensus.cc:738] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.567158 1171 consensus_queue.cc:260] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.567806 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (table default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916]). Expected version 5 got 0
I20250625 13:52:14.568022 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
I20250625 13:52:14.569205 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6b6fcfe7007d45478310c91778e8fdff" } modify_external_catalogs: true
I20250625 13:52:14.569459 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6b6fcfe7007d45478310c91778e8fdff" } modify_external_catalogs: true
I20250625 13:52:14.570058 1171 ts_tablet_manager.cc:1428] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:14.570401 1171 tablet_bootstrap.cc:492] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.575765 1171 tablet_bootstrap.cc:654] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:14.576599 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 669de0b7c93a4efea8623dab2e462c8d (table default.Table_8315e81e0d0843819735795f79fa91fb [id=d3092ac7d43a4e9891fe289b304e937c], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 669de0b7c93a4efea8623dab2e462c8d consensus state has no leader
I20250625 13:52:14.495836 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43ea6e00a7d40baa9e849a59f0bd42b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.538081 2371 tablet.cc:1722] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 13:52:14.544431 2358 raft_consensus.cc:491] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.579854 2358 raft_consensus.cc:513] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.580521 2358 leader_election.cc:290] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.580818 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f56a8a6cb964d3a979e2f1d944eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.581182 2380 leader_election.cc:290] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.581369 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a2007e51d424053bc2836dedf46add2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:14.581535 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f56a8a6cb964d3a979e2f1d944eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.581132 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a2007e51d424053bc2836dedf46add2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:14.581892 711 leader_election.cc:343] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.582425 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87f5b36ddb474ed6903c14876b236ba8" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:14.582587 913 raft_consensus.cc:3058] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:14.583493 711 leader_election.cc:343] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.583604 711 leader_election.cc:304] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:14.584004 2358 raft_consensus.cc:2747] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.586109 913 raft_consensus.cc:2466] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:14.586614 983 leader_election.cc:304] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.586962 2380 raft_consensus.cc:2802] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:14.587160 2380 raft_consensus.cc:695] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:14.587536 2380 consensus_queue.cc:237] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.579437 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f56a8a6cb964d3a979e2f1d944eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.591451 1171 tablet_bootstrap.cc:492] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.591847 1171 ts_tablet_manager.cc:1397] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20250625 13:52:14.592603 1171 raft_consensus.cc:357] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.593256 1171 raft_consensus.cc:383] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.593374 1171 raft_consensus.cc:738] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.593659 1171 consensus_queue.cc:260] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.597348 1171 ts_tablet_manager.cc:1428] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 13:52:14.597657 1171 tablet_bootstrap.cc:492] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.579416 1165 log.cc:1199] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/a30a35e3639a43c5b078169060824723
I20250625 13:52:14.598583 1165 ts_tablet_manager.cc:1939] T a30a35e3639a43c5b078169060824723 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
W20250625 13:52:14.600068 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9def3e1e149a4045a1bb2da76ced8eaf consensus state has no leader
I20250625 13:52:14.602854 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a85c5aeb7cd742e1ad82b20001c5c2cc" } new_table_name: "default.Table_ae093472c1b64d16bdfd6fd142faca11" modify_external_catalogs: true
I20250625 13:52:14.603704 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet a30a35e3639a43c5b078169060824723 (table default.Table_d7faba35e83944d586e1e53d4b524950 [id=33400d976688476eb4450525402f12be]) successfully deleted
W20250625 13:52:14.603986 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d7f68a0c4bb341a9a90cc2be0f78daca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7f68a0c4bb341a9a90cc2be0f78daca already in progress: creating tablet
I20250625 13:52:14.608040 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bb9fcf91fa304a359213a3e5267a9cac
I20250625 13:52:14.608297 594 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet e7d263c0d3824664a434b336f132ebb3
I20250625 13:52:14.609242 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet bb9fcf91fa304a359213a3e5267a9cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.609531 879 tablet_service.cc:1515] Processing DeleteTablet for tablet bb9fcf91fa304a359213a3e5267a9cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:14.610116 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6f4712c89364336820178423f4301a8 consensus state has no leader
I20250625 13:52:14.610322 1171 tablet_bootstrap.cc:654] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.610728 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 83699efa521a44f092cb738002aa58e4
I20250625 13:52:14.612633 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0346ed4f8d3f46aa84cbabbcadcebf23
I20250625 13:52:14.613344 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.613654 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 83699efa521a44f092cb738002aa58e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.613919 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.614167 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 83699efa521a44f092cb738002aa58e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.616808 592 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916] alter complete (version 5)
I20250625 13:52:14.618144 1171 tablet_bootstrap.cc:492] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.618356 1171 ts_tablet_manager.cc:1397] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.007s
I20250625 13:52:14.618914 744 tablet_service.cc:1515] Processing DeleteTablet for tablet bb9fcf91fa304a359213a3e5267a9cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.619266 1171 raft_consensus.cc:357] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.619511 1171 raft_consensus.cc:383] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.619637 1171 raft_consensus.cc:738] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.619773 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 83699efa521a44f092cb738002aa58e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.619984 1171 consensus_queue.cc:260] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.620411 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.623162 1168 ts_tablet_manager.cc:1905] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:14.628808 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5592ab667b04932896addf7514af24e consensus state has no leader
I20250625 13:52:14.629462 1171 ts_tablet_manager.cc:1428] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 13:52:14.455859 2254 raft_consensus.cc:2241] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.634239 1171 tablet_bootstrap.cc:492] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.634249 2254 raft_consensus.cc:2270] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.634928 2254 ts_tablet_manager.cc:1905] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:14.636180 1171 tablet_bootstrap.cc:654] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.643961 1171 tablet_bootstrap.cc:492] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.644181 1171 ts_tablet_manager.cc:1397] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20250625 13:52:14.645057 1171 raft_consensus.cc:357] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.645306 1171 raft_consensus.cc:383] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.645397 1171 raft_consensus.cc:738] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.645678 1171 consensus_queue.cc:260] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.646281 1171 ts_tablet_manager.cc:1428] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:14.646595 1171 tablet_bootstrap.cc:492] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.648007 1171 tablet_bootstrap.cc:654] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.655844 1171 tablet_bootstrap.cc:492] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.656100 1171 ts_tablet_manager.cc:1397] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250625 13:52:14.657424 1171 raft_consensus.cc:357] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.657666 1171 raft_consensus.cc:383] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.657759 1171 raft_consensus.cc:738] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.658028 1171 consensus_queue.cc:260] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.658718 1171 ts_tablet_manager.cc:1428] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:14.659044 1171 tablet_bootstrap.cc:492] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.660413 1171 tablet_bootstrap.cc:654] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.660396 1165 tablet_replica.cc:331] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:14.660667 1165 raft_consensus.cc:2241] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.660864 1165 raft_consensus.cc:2270] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.661509 1165 ts_tablet_manager.cc:1905] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:14.668217 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet fc9a79f066e84a7b87439ea4516f0287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9a79f066e84a7b87439ea4516f0287 already in progress: creating tablet
W20250625 13:52:14.668602 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet fc9a79f066e84a7b87439ea4516f0287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9a79f066e84a7b87439ea4516f0287 already in progress: creating tablet
I20250625 13:52:14.669719 1166 tablet_bootstrap.cc:492] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:14.669916 1166 ts_tablet_manager.cc:1397] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.317s user 0.001s sys 0.023s
I20250625 13:52:14.670676 1170 tablet_bootstrap.cc:492] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:14.670792 1166 raft_consensus.cc:357] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.670972 1166 raft_consensus.cc:383] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.671030 1166 raft_consensus.cc:738] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.670917 1170 ts_tablet_manager.cc:1397] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.337s user 0.008s sys 0.031s
I20250625 13:52:14.671232 1166 consensus_queue.cc:260] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.671783 1166 ts_tablet_manager.cc:1428] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:14.673117 1170 raft_consensus.cc:357] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.673525 1170 raft_consensus.cc:383] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.673686 1170 raft_consensus.cc:738] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.674124 1170 consensus_queue.cc:260] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.675104 1170 ts_tablet_manager.cc:1428] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:14.677065 1170 tablet_bootstrap.cc:492] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:14.682514 1171 tablet_bootstrap.cc:492] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:14.682675 1171 ts_tablet_manager.cc:1397] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20250625 13:52:14.685472 1171 raft_consensus.cc:357] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.685683 1171 raft_consensus.cc:383] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.685782 1171 raft_consensus.cc:738] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.685995 1171 consensus_queue.cc:260] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.686604 1171 ts_tablet_manager.cc:1428] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250625 13:52:14.686600 2348 raft_consensus.cc:695] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:14.686918 1171 tablet_bootstrap.cc:492] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:14.687381 1166 tablet_bootstrap.cc:492] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:14.689627 1166 tablet_bootstrap.cc:654] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.695807 1168 ts_tablet_manager.cc:1918] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:14.695969 1168 log.cc:1199] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a0e81d4c904248eba858462bcf31c5d9
I20250625 13:52:14.696317 1168 ts_tablet_manager.cc:1939] T a0e81d4c904248eba858462bcf31c5d9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:14.686936 2348 consensus_queue.cc:237] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.703467 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 9710609da88d4b42993b3aef02416543 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:14.704516 1171 tablet_bootstrap.cc:654] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.707839 1165 ts_tablet_manager.cc:1918] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:14.711073 2254 ts_tablet_manager.cc:1918] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.711279 2254 log.cc:1199] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/9c1deb52366c404cbdc08076d4bcb3d1
W20250625 13:52:14.713826 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 67682.2us
W20250625 13:52:14.728708 851 consensus_peers.cc:487] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:14.739455 2338 raft_consensus.cc:491] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.739706 2338 raft_consensus.cc:513] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.740352 2338 leader_election.cc:290] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:14.742138 851 leader_election.cc:343] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 37e4bd4aaab143ffa65a668a952ae345
I20250625 13:52:14.742638 2210 raft_consensus.cc:491] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.742792 2210 raft_consensus.cc:513] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.743600 2210 leader_election.cc:290] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.743754 2210 raft_consensus.cc:491] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.743866 2210 raft_consensus.cc:513] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.744431 2210 leader_election.cc:290] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.746194 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
I20250625 13:52:14.746253 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:14.748358 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.750982 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5a0c4d531f874f9b838780f22299af09 (DEFAULT_TABLE table=default.Table_bf56f0c20508449b8c381ba2c3510578 [id=805422dee28646dab3461b2738adfb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.751433 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0c4d531f874f9b838780f22299af09. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:14.755079 851 leader_election.cc:343] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 723246e1e6824faf9f9d39c3ea941fe4
I20250625 13:52:14.757208 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9710609da88d4b42993b3aef02416543
I20250625 13:52:14.757270 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
I20250625 13:52:14.757671 592 catalog_manager.cc:6652] ReplaceTablet: tablet 9710609da88d4b42993b3aef02416543 deleted and replaced by tablet 724a36167dc64f1789cbf736cede07df
I20250625 13:52:14.758351 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 9710609da88d4b42993b3aef02416543 with delete_type TABLET_DATA_DELETED (replaced by tablet 724a36167dc64f1789cbf736cede07df) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.760807 879 tablet_service.cc:1468] Processing CreateTablet for tablet 8c4082d17b5842f3bf7ae27fc9c1b3b3 (DEFAULT_TABLE table=default.Table_03fee4b89d0c4186beb0de611b2fc05f [id=286bb158e3de4eb79bc1f6c9bd117ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.761267 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4082d17b5842f3bf7ae27fc9c1b3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.761317 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.761502 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_54c3accf07ed49259aa5cfdfcd715f72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.762054 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54c3accf07ed49259aa5cfdfcd715f72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.763445 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c00510ec28ac4833831eb7a04cb5192f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.764014 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c00510ec28ac4833831eb7a04cb5192f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.764946 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_603f8675b541481380889a4e72b78dbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.765336 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_603f8675b541481380889a4e72b78dbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.766088 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.766901 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4215c08737a746b59dd7eb42a6a34ba5" } modify_external_catalogs: true
I20250625 13:52:14.767068 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4215c08737a746b59dd7eb42a6a34ba5" } modify_external_catalogs: true
I20250625 13:52:14.767326 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 3859f399a7b84de99658380bd6c24ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.775040 878 tablet_service.cc:1468] Processing CreateTablet for tablet 259dc5d434ff43bc85fee5b3edc45064 (DEFAULT_TABLE table=default.Table_f7f62312f5ed44fa821485933ffc5d0b [id=439c4d4ad58c40faad7669c9c58218af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.775599 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259dc5d434ff43bc85fee5b3edc45064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.775831 886 tablet_service.cc:1468] Processing CreateTablet for tablet cdb955e3410945f898639235cd25bad7 (DEFAULT_TABLE table=default.Table_652e0ff5dc3f4eb286b425ec0630c138 [id=48ae56f42fe14313ba05232f01f825bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.776206 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb955e3410945f898639235cd25bad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.777644 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bbd3200bd1384187978aeb7089ec4e03
I20250625 13:52:14.778663 894 tablet_service.cc:1515] Processing DeleteTablet for tablet bbd3200bd1384187978aeb7089ec4e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.780772 894 tablet_service.cc:1468] Processing CreateTablet for tablet f836a691aaf8463fb69bcf9c98aaadda (DEFAULT_TABLE table=default.Table_9c3c0c15b51d4dca8c18dfb78f2054d5 [id=02c7e2be40284b18af9258ccd83a7e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.781093 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f836a691aaf8463fb69bcf9c98aaadda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.796773 882 tablet_service.cc:1468] Processing CreateTablet for tablet 3adc62e0d4cd419980a9e1296c3ea32d (DEFAULT_TABLE table=default.Table_7ce4c37b46af4972a39f3bd6c431af1e [id=ff21d4c3ff4c4fafbd9b345be57fc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.797222 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adc62e0d4cd419980a9e1296c3ea32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.696655 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 910) took 4611 ms. Trace:
I20250625 13:52:14.803427 1168 rpcz_store.cc:276] 0625 13:52:10.085573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.085628 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:14.417057 (+4331429us) ts_tablet_manager.cc:1145] Deleting tablet a0e81d4c904248eba858462bcf31c5d9
0625 13:52:14.417065 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.626320 (+209255us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.695693 (+ 69373us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.696641 (+ 948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4331207}
I20250625 13:52:14.803912 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet a0e81d4c904248eba858462bcf31c5d9 (table default.Table_d7e04ac5a7b3491bb209bc40967a3059 [id=ff6be4b24d8c4b6e8b9f3784067593a5]) successfully deleted
I20250625 13:52:14.804514 1168 tablet_replica.cc:331] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:14.804946 2338 raft_consensus.cc:491] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.805107 1168 raft_consensus.cc:2241] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:14.805142 2338 raft_consensus.cc:513] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.805260 1168 raft_consensus.cc:2270] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.805775 1168 ts_tablet_manager.cc:1905] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:14.805819 2338 leader_election.cc:290] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.806164 894 tablet_service.cc:1468] Processing CreateTablet for tablet b42693af8f214e34b0326e73b4401f1a (DEFAULT_TABLE table=default.Table_5e8ec384f9c44f70a2494253304496f4 [id=700f7cc48c3b4cf2a9e7b06e570fc32b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.806519 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42693af8f214e34b0326e73b4401f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.807592 886 tablet_service.cc:1468] Processing CreateTablet for tablet 25701484036a49dea0fb600c4dd2e998 (DEFAULT_TABLE table=default.Table_45c7b1313316471f9fc8a44c2f19554b [id=42a1c7c209384e59aa37028ffbcd2a41]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.807809 851 leader_election.cc:343] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 60ef50a3624b4c79acc803ff0e2ed364
I20250625 13:52:14.807982 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25701484036a49dea0fb600c4dd2e998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.712836 1170 tablet_bootstrap.cc:654] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.722451 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 024f6cb363594420b0d5480fa79d156b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:08 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.741609 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37e4bd4aaab143ffa65a668a952ae345" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:14.741962 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 042359eacaa2451abc0fa6ed447a2905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:09 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.745450 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723246e1e6824faf9f9d39c3ea941fe4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:14.773454 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:14.774718 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 5a0c4d531f874f9b838780f22299af09 (DEFAULT_TABLE table=default.Table_bf56f0c20508449b8c381ba2c3510578 [id=805422dee28646dab3461b2738adfb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.783146 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 9710609da88d4b42993b3aef02416543 with delete_type TABLET_DATA_DELETED (replaced by tablet 724a36167dc64f1789cbf736cede07df) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.783470 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 8c4082d17b5842f3bf7ae27fc9c1b3b3 (DEFAULT_TABLE table=default.Table_03fee4b89d0c4186beb0de611b2fc05f [id=286bb158e3de4eb79bc1f6c9bd117ae4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.812923 849 leader_election.cc:343] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.813000 2254 ts_tablet_manager.cc:1939] T 9c1deb52366c404cbdc08076d4bcb3d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:14.783807 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 259dc5d434ff43bc85fee5b3edc45064 (DEFAULT_TABLE table=default.Table_f7f62312f5ed44fa821485933ffc5d0b [id=439c4d4ad58c40faad7669c9c58218af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.813047 849 leader_election.cc:304] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.784001 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet bbd3200bd1384187978aeb7089ec4e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:14.813506 849 leader_election.cc:343] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.813622 849 leader_election.cc:304] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:14.814023 849 leader_election.cc:343] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.784291 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f836a691aaf8463fb69bcf9c98aaadda (DEFAULT_TABLE table=default.Table_9c3c0c15b51d4dca8c18dfb78f2054d5 [id=02c7e2be40284b18af9258ccd83a7e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.785645 1015 tablet_service.cc:1468] Processing CreateTablet for tablet cdb955e3410945f898639235cd25bad7 (DEFAULT_TABLE table=default.Table_652e0ff5dc3f4eb286b425ec0630c138 [id=48ae56f42fe14313ba05232f01f825bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.814385 1170 tablet_bootstrap.cc:492] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:14.794795 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 3adc62e0d4cd419980a9e1296c3ea32d (DEFAULT_TABLE table=default.Table_7ce4c37b46af4972a39f3bd6c431af1e [id=ff21d4c3ff4c4fafbd9b345be57fc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.814419 2271 raft_consensus.cc:2747] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.814536 1170 ts_tablet_manager.cc:1397] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.001s
I20250625 13:52:14.803701 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 25701484036a49dea0fb600c4dd2e998 (DEFAULT_TABLE table=default.Table_45c7b1313316471f9fc8a44c2f19554b [id=42a1c7c209384e59aa37028ffbcd2a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.814597 2271 raft_consensus.cc:2747] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.804968 1016 tablet_service.cc:1468] Processing CreateTablet for tablet b42693af8f214e34b0326e73b4401f1a (DEFAULT_TABLE table=default.Table_5e8ec384f9c44f70a2494253304496f4 [id=700f7cc48c3b4cf2a9e7b06e570fc32b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.815379 1170 raft_consensus.cc:357] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.815572 1170 raft_consensus.cc:383] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.815657 1170 raft_consensus.cc:738] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.815894 1170 consensus_queue.cc:260] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.816392 1170 ts_tablet_manager.cc:1428] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:14.816700 1170 tablet_bootstrap.cc:492] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:14.816913 879 tablet_service.cc:1468] Processing CreateTablet for tablet d92fe17f576e491eb189a10ca72a7e09 (DEFAULT_TABLE table=default.Table_51a539bc437b40f6ae158307f8306a0f [id=af749fdb0a084e269160c85df886d1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.817234 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92fe17f576e491eb189a10ca72a7e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.818948 1170 tablet_bootstrap.cc:654] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.806157 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d92fe17f576e491eb189a10ca72a7e09 (DEFAULT_TABLE table=default.Table_51a539bc437b40f6ae158307f8306a0f [id=af749fdb0a084e269160c85df886d1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.806674 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60ef50a3624b4c79acc803ff0e2ed364" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:14.820070 849 leader_election.cc:343] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.820175 849 leader_election.cc:304] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:14.820226 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0c4d531f874f9b838780f22299af09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.820581 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25701484036a49dea0fb600c4dd2e998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.820928 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4082d17b5842f3bf7ae27fc9c1b3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.823436 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259dc5d434ff43bc85fee5b3edc45064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.823566 2338 raft_consensus.cc:2747] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.823858 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adc62e0d4cd419980a9e1296c3ea32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.824782 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42693af8f214e34b0326e73b4401f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.824926 890 tablet_service.cc:1468] Processing CreateTablet for tablet 0cf35321b38a4e81b41563f711dbb86d (DEFAULT_TABLE table=default.Table_cf8d3ee817784b2aa014fc65c502b75c [id=3304d34570eb45f48d769f1a0cf2e3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.825276 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf35321b38a4e81b41563f711dbb86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.827421 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f836a691aaf8463fb69bcf9c98aaadda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.827618 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9f53fde8e6ee42d5a18d464c7eac4efe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.828176 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f53fde8e6ee42d5a18d464c7eac4efe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.830149 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0ec681dfabc544efa1c578f0af675bb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.830638 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ec681dfabc544efa1c578f0af675bb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.831925 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_51a2a2b4189c46bca8e8c16bb0b08d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.832398 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51a2a2b4189c46bca8e8c16bb0b08d10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.832628 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb955e3410945f898639235cd25bad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.833559 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_46ba401738b64322afb12c3635116d5c" } new_table_name: "default.Table_3a59bba14f7b466ca1b54de40428bdb3" modify_external_catalogs: true
I20250625 13:52:14.836367 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 0cf35321b38a4e81b41563f711dbb86d (DEFAULT_TABLE table=default.Table_cf8d3ee817784b2aa014fc65c502b75c [id=3304d34570eb45f48d769f1a0cf2e3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.844367 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7e1e10c14c834d1ea94c3310c244e9d6" } new_table_name: "default.Table_e7d5b0a0e4d64a318b47d1476ac90f2b" modify_external_catalogs: true
I20250625 13:52:14.844622 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_330a9a9673154b0fb697d5f2cc16a4ab" } modify_external_catalogs: true
I20250625 13:52:14.844852 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_330a9a9673154b0fb697d5f2cc16a4ab" } modify_external_catalogs: true
I20250625 13:52:14.845148 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_39d5b8799785461d861d7d1ef1fba8e2" } modify_external_catalogs: true
I20250625 13:52:14.845300 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_39d5b8799785461d861d7d1ef1fba8e2" } modify_external_catalogs: true
I20250625 13:52:14.845768 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b88679cb7cc74587927204b82e5915f4" } modify_external_catalogs: true
I20250625 13:52:14.845945 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b88679cb7cc74587927204b82e5915f4" } modify_external_catalogs: true
I20250625 13:52:14.851408 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ade61b1fec8c4d09a698917fe4a50bba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.852185 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ade61b1fec8c4d09a698917fe4a50bba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.852527 1022 tablet_service.cc:1468] Processing CreateTablet for tablet b195249fb8274a7cb8d8b9e8423720ae (DEFAULT_TABLE table=default.Table_42800fa616864465b16c7cb54ad50494 [id=b259c537c4614fa29b292c6565e168d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.853410 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_177c0af3b570495e804887f8eff6a8d9" } new_table_name: "default.Table_54179fba308a4f9bb072bf4ffd9fe873" modify_external_catalogs: true
I20250625 13:52:14.856114 890 tablet_service.cc:1468] Processing CreateTablet for tablet b195249fb8274a7cb8d8b9e8423720ae (DEFAULT_TABLE table=default.Table_42800fa616864465b16c7cb54ad50494 [id=b259c537c4614fa29b292c6565e168d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.856593 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b195249fb8274a7cb8d8b9e8423720ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.857211 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b584197acb904919a3681da75860d445" } modify_external_catalogs: true
I20250625 13:52:14.857401 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b584197acb904919a3681da75860d445" } modify_external_catalogs: true
I20250625 13:52:14.860414 1014 tablet_service.cc:1468] Processing CreateTablet for tablet ae58895707cb4542ba2b4e6f60a82f93 (DEFAULT_TABLE table=default.Table_78e65c78bd4d43f0900060ea0aa8203a [id=479b215e18984958bdd49fe60c2fc6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.861536 896 tablet_service.cc:1468] Processing CreateTablet for tablet ae58895707cb4542ba2b4e6f60a82f93 (DEFAULT_TABLE table=default.Table_78e65c78bd4d43f0900060ea0aa8203a [id=479b215e18984958bdd49fe60c2fc6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.865224 891 tablet_service.cc:1468] Processing CreateTablet for tablet 09733b6641dd49c9b29633af1c466aa4 (DEFAULT_TABLE table=default.Table_f3b3e3e07eb84bfa8a9396b3161a9114 [id=5227457618334dd0ad007356f6a79539]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.867569 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62822a0a8bf54f0184312ddddfb561aa consensus state has no leader
W20250625 13:52:14.867683 849 consensus_peers.cc:487] T 86af0ea6648045a1ac4ba903d20849b7 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20250625 13:52:14.867956 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cfe03d5988ab4502a2f04e1eef206474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.868724 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d2e8f4e3b45a4065b74485157f31247d
I20250625 13:52:14.868875 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0bc136dc84de44b49abcdf4100a86137
I20250625 13:52:14.869369 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9f23f34708854611a2bc13f3b171ee8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:14.869837 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f23f34708854611a2bc13f3b171ee8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:14.868453 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfe03d5988ab4502a2f04e1eef206474 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:14.870349 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fa52e79e00c4f9c97a05c353e11ae2c consensus state has no leader
I20250625 13:52:14.870452 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 29e223675c7842b383865a270b97ce58
I20250625 13:52:14.872942 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3e78c988506748b8b316cc58f5c6a9dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:14.873386 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d2e8f4e3b45a4065b74485157f31247d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:14.873476 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e78c988506748b8b316cc58f5c6a9dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:14.873633 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 29e223675c7842b383865a270b97ce58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.873800 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bc136dc84de44b49abcdf4100a86137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.874421 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d2e8f4e3b45a4065b74485157f31247d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.874673 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 29e223675c7842b383865a270b97ce58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.874663 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_639d5431f288425b907989942e4aa7f5" } new_table_name: "default.Table_2f58650bfd4a431da4bcc8f3331ee518" modify_external_catalogs: true
I20250625 13:52:14.875066 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bc136dc84de44b49abcdf4100a86137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.876108 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3f882e0fab474cf2aef02b0067d704f8" } new_table_name: "default.Table_46e5b7c336504c6a87427975a0168a63" modify_external_catalogs: true
I20250625 13:52:14.883113 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae58895707cb4542ba2b4e6f60a82f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.884126 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09733b6641dd49c9b29633af1c466aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.885095 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1b2914ce9b4c40a0bc00d0dd5ac72221
I20250625 13:52:14.885677 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 09733b6641dd49c9b29633af1c466aa4 (DEFAULT_TABLE table=default.Table_f3b3e3e07eb84bfa8a9396b3161a9114 [id=5227457618334dd0ad007356f6a79539]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:14.886386 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b387e93b38754e31999ec1f2ae90d626 consensus state has no leader
I20250625 13:52:14.886426 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:14.887871 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 4e4b584d5a6447da9c06859a1cb1030e (DEFAULT_TABLE table=default.Table_6afb92085aaa47bd961481fae3d17266 [id=f50db1e3d5e84d7dbc91f3c4683e3f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.888744 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:14.889374 889 tablet_service.cc:1468] Processing CreateTablet for tablet 4e4b584d5a6447da9c06859a1cb1030e (DEFAULT_TABLE table=default.Table_6afb92085aaa47bd961481fae3d17266 [id=f50db1e3d5e84d7dbc91f3c4683e3f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.889762 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4b584d5a6447da9c06859a1cb1030e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.748347 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:14.757974 753 tablet_service.cc:1468] Processing CreateTablet for tablet 5a0c4d531f874f9b838780f22299af09 (DEFAULT_TABLE table=default.Table_bf56f0c20508449b8c381ba2c3510578 [id=805422dee28646dab3461b2738adfb85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.759557 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 9710609da88d4b42993b3aef02416543 with delete_type TABLET_DATA_DELETED (replaced by tablet 724a36167dc64f1789cbf736cede07df) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.760215 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8c4082d17b5842f3bf7ae27fc9c1b3b3 (DEFAULT_TABLE table=default.Table_03fee4b89d0c4186beb0de611b2fc05f [id=286bb158e3de4eb79bc1f6c9bd117ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.766199 2358 raft_consensus.cc:491] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.775841 757 tablet_service.cc:1468] Processing CreateTablet for tablet 259dc5d434ff43bc85fee5b3edc45064 (DEFAULT_TABLE table=default.Table_f7f62312f5ed44fa821485933ffc5d0b [id=439c4d4ad58c40faad7669c9c58218af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.776427 739 tablet_service.cc:1468] Processing CreateTablet for tablet cdb955e3410945f898639235cd25bad7 (DEFAULT_TABLE table=default.Table_652e0ff5dc3f4eb286b425ec0630c138 [id=48ae56f42fe14313ba05232f01f825bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.779487 755 tablet_service.cc:1515] Processing DeleteTablet for tablet bbd3200bd1384187978aeb7089ec4e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.780305 756 tablet_service.cc:1468] Processing CreateTablet for tablet f836a691aaf8463fb69bcf9c98aaadda (DEFAULT_TABLE table=default.Table_9c3c0c15b51d4dca8c18dfb78f2054d5 [id=02c7e2be40284b18af9258ccd83a7e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.816411 740 tablet_service.cc:1468] Processing CreateTablet for tablet 3adc62e0d4cd419980a9e1296c3ea32d (DEFAULT_TABLE table=default.Table_7ce4c37b46af4972a39f3bd6c431af1e [id=ff21d4c3ff4c4fafbd9b345be57fc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.816526 746 tablet_service.cc:1468] Processing CreateTablet for tablet b42693af8f214e34b0326e73b4401f1a (DEFAULT_TABLE table=default.Table_5e8ec384f9c44f70a2494253304496f4 [id=700f7cc48c3b4cf2a9e7b06e570fc32b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.816718 758 tablet_service.cc:1468] Processing CreateTablet for tablet 25701484036a49dea0fb600c4dd2e998 (DEFAULT_TABLE table=default.Table_45c7b1313316471f9fc8a44c2f19554b [id=42a1c7c209384e59aa37028ffbcd2a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.816869 748 tablet_service.cc:1468] Processing CreateTablet for tablet d92fe17f576e491eb189a10ca72a7e09 (DEFAULT_TABLE table=default.Table_51a539bc437b40f6ae158307f8306a0f [id=af749fdb0a084e269160c85df886d1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.893584 1165 log.cc:1199] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/39ac79102eab46688e767faed6d6f114
I20250625 13:52:14.893627 775 raft_consensus.cc:2466] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:14.836887 750 tablet_service.cc:1468] Processing CreateTablet for tablet 0cf35321b38a4e81b41563f711dbb86d (DEFAULT_TABLE table=default.Table_cf8d3ee817784b2aa014fc65c502b75c [id=3304d34570eb45f48d769f1a0cf2e3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.855558 747 tablet_service.cc:1468] Processing CreateTablet for tablet b195249fb8274a7cb8d8b9e8423720ae (DEFAULT_TABLE table=default.Table_42800fa616864465b16c7cb54ad50494 [id=b259c537c4614fa29b292c6565e168d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.862284 745 tablet_service.cc:1468] Processing CreateTablet for tablet ae58895707cb4542ba2b4e6f60a82f93 (DEFAULT_TABLE table=default.Table_78e65c78bd4d43f0900060ea0aa8203a [id=479b215e18984958bdd49fe60c2fc6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.866108 742 tablet_service.cc:1468] Processing CreateTablet for tablet 09733b6641dd49c9b29633af1c466aa4 (DEFAULT_TABLE table=default.Table_f3b3e3e07eb84bfa8a9396b3161a9114 [id=5227457618334dd0ad007356f6a79539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.879513 754 tablet_service.cc:1515] Processing DeleteTablet for tablet d2e8f4e3b45a4065b74485157f31247d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.879607 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 29e223675c7842b383865a270b97ce58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.896430 851 leader_election.cc:304] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:14.894043 2358 raft_consensus.cc:513] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.896934 2338 raft_consensus.cc:2802] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:14.897012 2338 raft_consensus.cc:491] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:14.897073 2338 raft_consensus.cc:3058] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:14.897080 2358 leader_election.cc:290] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.897188 2358 raft_consensus.cc:491] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:14.897317 2358 raft_consensus.cc:513] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.817342 1166 tablet_bootstrap.cc:492] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:14.888600 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.889060 749 tablet_service.cc:1468] Processing CreateTablet for tablet 4e4b584d5a6447da9c06859a1cb1030e (DEFAULT_TABLE table=default.Table_6afb92085aaa47bd961481fae3d17266 [id=f50db1e3d5e84d7dbc91f3c4683e3f53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.879709 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bc136dc84de44b49abcdf4100a86137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:14 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:14.897739 1166 ts_tablet_manager.cc:1397] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.210s user 0.004s sys 0.001s
I20250625 13:52:14.898947 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:14.899415 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dec08d20a09444f5b9a102a8712e36c7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:14.900168 711 leader_election.cc:343] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:14.900350 711 leader_election.cc:343] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.897811 2358 leader_election.cc:290] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:14.893987 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0c4d531f874f9b838780f22299af09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.901854 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09733b6641dd49c9b29633af1c466aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.902069 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb955e3410945f898639235cd25bad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.903434 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adc62e0d4cd419980a9e1296c3ea32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.906011 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b195249fb8274a7cb8d8b9e8423720ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.907413 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4082d17b5842f3bf7ae27fc9c1b3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.907583 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25701484036a49dea0fb600c4dd2e998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.907723 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259dc5d434ff43bc85fee5b3edc45064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.907862 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae58895707cb4542ba2b4e6f60a82f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.911412 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f836a691aaf8463fb69bcf9c98aaadda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.911594 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf35321b38a4e81b41563f711dbb86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.911792 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4b584d5a6447da9c06859a1cb1030e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.912002 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42693af8f214e34b0326e73b4401f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.916100 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92fe17f576e491eb189a10ca72a7e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.918689 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:14.919205 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dec08d20a09444f5b9a102a8712e36c7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:14.919634 711 leader_election.cc:343] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.919739 711 leader_election.cc:304] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:14.920305 711 leader_election.cc:343] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.920397 711 leader_election.cc:304] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:14.920799 2358 raft_consensus.cc:2747] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.920960 2358 raft_consensus.cc:2747] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:14.923566 1165 ts_tablet_manager.cc:1939] T 39ac79102eab46688e767faed6d6f114 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:14.924351 1166 raft_consensus.cc:357] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.924541 1166 raft_consensus.cc:383] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:14.924609 1166 raft_consensus.cc:738] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:14.924829 1166 consensus_queue.cc:260] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.925374 1166 ts_tablet_manager.cc:1428] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250625 13:52:14.947567 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 39ac79102eab46688e767faed6d6f114 (table default.Table_f26987562c25452388d457eb3b600595 [id=62553d70f8ee470f944206ddc8cc2089]) successfully deleted
I20250625 13:52:14.953938 1166 tablet_bootstrap.cc:492] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:14.955993 1166 tablet_bootstrap.cc:654] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:14.957165 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92fe17f576e491eb189a10ca72a7e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.958508 1165 tablet_replica.cc:331] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:14.958992 1165 raft_consensus.cc:2241] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:14.959411 1165 raft_consensus.cc:2270] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:14.961315 894 tablet_service.cc:1468] Processing CreateTablet for tablet 9700337fd5984819a8b2f16b28c2ccac (DEFAULT_TABLE table=default.Table_18f5eaac4ead4aa8bb9e663f4fec5d79 [id=c85b0b15adcb40e6af12995792c11b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.961792 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9700337fd5984819a8b2f16b28c2ccac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.962962 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 9700337fd5984819a8b2f16b28c2ccac (DEFAULT_TABLE table=default.Table_18f5eaac4ead4aa8bb9e663f4fec5d79 [id=c85b0b15adcb40e6af12995792c11b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.963063 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 42e9f5736d8a43039eccaf6120414d48 (DEFAULT_TABLE table=default.Table_6dc8600ee239471dbe4e422ce34db986 [id=2cb3ccc7134642fe9747bd432d8609d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.966146 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf35321b38a4e81b41563f711dbb86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.966463 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b195249fb8274a7cb8d8b9e8423720ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.968047 1168 ts_tablet_manager.cc:1918] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:14.968150 1168 log.cc:1199] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ac4d368347444846880954ced9651404
W20250625 13:52:14.969676 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d0174345facc4216b6ddeb8f8c73790d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0174345facc4216b6ddeb8f8c73790d already in progress: creating tablet
W20250625 13:52:14.970388 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3562e8b8731a4d76bb1a36189f957e07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3562e8b8731a4d76bb1a36189f957e07 already in progress: creating tablet
W20250625 13:52:14.970567 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8f88091205b944b386094e50cf92c36b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f88091205b944b386094e50cf92c36b already in progress: creating tablet
W20250625 13:52:14.970741 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 529c3bf8e72f46948cfb864445ff51f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 529c3bf8e72f46948cfb864445ff51f3 already in progress: creating tablet
I20250625 13:52:14.971855 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae58895707cb4542ba2b4e6f60a82f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.971773 758 tablet_service.cc:1468] Processing CreateTablet for tablet 9700337fd5984819a8b2f16b28c2ccac (DEFAULT_TABLE table=default.Table_18f5eaac4ead4aa8bb9e663f4fec5d79 [id=c85b0b15adcb40e6af12995792c11b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.971925 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 755f8b4bfb844afba8579c8289ace460 (DEFAULT_TABLE table=default.Table_22c63d145f8844b286282fdf48f6c32c [id=b2d5ab16b1f44523bd0cd7ff6f1d041f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.972143 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9700337fd5984819a8b2f16b28c2ccac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.972270 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755f8b4bfb844afba8579c8289ace460. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.974221 2338 raft_consensus.cc:513] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:14.974840 2338 leader_election.cc:290] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:14.975389 1025 tablet_service.cc:1468] Processing CreateTablet for tablet b387e93b38754e31999ec1f2ae90d626 (DEFAULT_TABLE table=default.Table_54179fba308a4f9bb072bf4ffd9fe873 [id=f722baa7fb9d41b9b1a4b0ce3b4b54fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.975613 746 tablet_service.cc:1468] Processing CreateTablet for tablet 42e9f5736d8a43039eccaf6120414d48 (DEFAULT_TABLE table=default.Table_6dc8600ee239471dbe4e422ce34db986 [id=2cb3ccc7134642fe9747bd432d8609d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.975728 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09733b6641dd49c9b29633af1c466aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.975934 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e9f5736d8a43039eccaf6120414d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.979573 757 tablet_service.cc:1468] Processing CreateTablet for tablet b387e93b38754e31999ec1f2ae90d626 (DEFAULT_TABLE table=default.Table_54179fba308a4f9bb072bf4ffd9fe873 [id=f722baa7fb9d41b9b1a4b0ce3b4b54fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.979911 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b387e93b38754e31999ec1f2ae90d626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.982860 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4b584d5a6447da9c06859a1cb1030e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.983580 750 tablet_service.cc:1468] Processing CreateTablet for tablet 755f8b4bfb844afba8579c8289ace460 (DEFAULT_TABLE table=default.Table_22c63d145f8844b286282fdf48f6c32c [id=b2d5ab16b1f44523bd0cd7ff6f1d041f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:14.983913 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755f8b4bfb844afba8579c8289ace460. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.988145 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:14.988688 849 leader_election.cc:343] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:14.989534 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1112) took 2141 ms. Trace:
I20250625 13:52:14.989719 2254 rpcz_store.cc:276] 0625 13:52:12.847723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.850927 (+ 3204us) service_pool.cc:225] Handling call
0625 13:52:14.455620 (+1604693us) ts_tablet_manager.cc:1145] Deleting tablet 9c1deb52366c404cbdc08076d4bcb3d1
0625 13:52:14.455629 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.638510 (+182881us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.710936 (+ 72426us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.989493 (+278557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1604419}
I20250625 13:52:14.990758 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9700337fd5984819a8b2f16b28c2ccac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.991003 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e9f5736d8a43039eccaf6120414d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:14.993868 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 9c1deb52366c404cbdc08076d4bcb3d1 (table default.Table_b2b0087eeea54823809b847648141d81 [id=253f3f5c5b8d4051800dc823b4ebaec9]) successfully deleted
I20250625 13:52:14.995481 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1113) took 2056 ms. Trace:
I20250625 13:52:14.995685 2254 rpcz_store.cc:276] 0625 13:52:12.938723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.939267 (+ 544us) service_pool.cc:225] Handling call
0625 13:52:14.990900 (+2051633us) ts_tablet_manager.cc:1145] Deleting tablet 466717307dd04d1cbdac9f03229f6a3b
0625 13:52:14.995340 (+ 4440us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.995438 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2051434}
I20250625 13:52:14.997429 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1127) took 1999 ms. Trace:
W20250625 13:52:14.997684 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 466717307dd04d1cbdac9f03229f6a3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 466717307dd04d1cbdac9f03229f6a3b already in progress: creating tablet
I20250625 13:52:14.998333 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b387e93b38754e31999ec1f2ae90d626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.003418 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 430c3bed360b4f32855a8adb99076608 with delete_type TABLET_DATA_DELETED (replaced by tablet 0f63e9037bd74f1eb39bd85aa820ec35) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.003479 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:15.003652 775 raft_consensus.cc:3058] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:15.006382 775 raft_consensus.cc:2466] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
W20250625 13:52:15.007433 851 consensus_peers.cc:487] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d95a376304e464ab75e2e1e00e7a924. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:15.007797 851 leader_election.cc:304] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:15.008208 2338 raft_consensus.cc:2802] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:15.008354 2338 raft_consensus.cc:695] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:15.008607 2338 consensus_queue.cc:237] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.008750 886 tablet_service.cc:1468] Processing CreateTablet for tablet 755f8b4bfb844afba8579c8289ace460 (DEFAULT_TABLE table=default.Table_22c63d145f8844b286282fdf48f6c32c [id=b2d5ab16b1f44523bd0cd7ff6f1d041f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.009509 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 755f8b4bfb844afba8579c8289ace460. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.009124 882 tablet_service.cc:1468] Processing CreateTablet for tablet 42e9f5736d8a43039eccaf6120414d48 (DEFAULT_TABLE table=default.Table_6dc8600ee239471dbe4e422ce34db986 [id=2cb3ccc7134642fe9747bd432d8609d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.011262 1170 tablet_bootstrap.cc:492] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:15.011492 1170 ts_tablet_manager.cc:1397] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.195s user 0.002s sys 0.032s
I20250625 13:52:15.011658 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e9f5736d8a43039eccaf6120414d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.012961 1170 raft_consensus.cc:357] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.013338 1170 raft_consensus.cc:383] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.013512 1170 raft_consensus.cc:738] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.013971 1170 consensus_queue.cc:260] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.019218 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c5fc63828d9408b9cf774a966f43cf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.020355 890 tablet_service.cc:1468] Processing CreateTablet for tablet b387e93b38754e31999ec1f2ae90d626 (DEFAULT_TABLE table=default.Table_54179fba308a4f9bb072bf4ffd9fe873 [id=f722baa7fb9d41b9b1a4b0ce3b4b54fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.020735 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b387e93b38754e31999ec1f2ae90d626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.025352 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_50fd1f26e7c343008610cf804b2a8d0a" } modify_external_catalogs: true
I20250625 13:52:15.025578 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_50fd1f26e7c343008610cf804b2a8d0a" } modify_external_catalogs: true
I20250625 13:52:15.026348 1171 tablet_bootstrap.cc:492] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:15.026587 1171 ts_tablet_manager.cc:1397] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.340s user 0.017s sys 0.039s
I20250625 13:52:15.026952 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f2a8252e5ba44d768968c5df61d58d21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.027526 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2a8252e5ba44d768968c5df61d58d21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.027501 1171 raft_consensus.cc:357] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.027657 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c52f47756979445e9c05f1f68d2283cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:15.027947 1171 raft_consensus.cc:383] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:52:15.028086 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c52f47756979445e9c05f1f68d2283cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.028090 1171 raft_consensus.cc:738] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.028561 1171 consensus_queue.cc:260] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.030887 1171 ts_tablet_manager.cc:1428] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:15.030925 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_685641e78e7d43b0a6432cd7f23aa34e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:15.031352 1171 tablet_bootstrap.cc:492] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:15.031641 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_685641e78e7d43b0a6432cd7f23aa34e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.032841 890 tablet_service.cc:1468] Processing CreateTablet for tablet 0c49eab755dd4f1cb16da4e10a24eecb (DEFAULT_TABLE table=default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179 [id=617f5dfcd64d408ea32281faf0fa6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.033233 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c49eab755dd4f1cb16da4e10a24eecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.034893 1170 ts_tablet_manager.cc:1428] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20250625 13:52:15.035458 1165 ts_tablet_manager.cc:1905] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.040319 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 0c49eab755dd4f1cb16da4e10a24eecb (DEFAULT_TABLE table=default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179 [id=617f5dfcd64d408ea32281faf0fa6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.040833 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c49eab755dd4f1cb16da4e10a24eecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.050057 1170 tablet_bootstrap.cc:492] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:15.055925 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a9969b2f1893446f9cec8338868abdfb (DEFAULT_TABLE table=default.Table_41a10889a8c045adb2961b8b2cde45c6 [id=c4fca0b2424b46c297de5330d494f5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.058792 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9969b2f1893446f9cec8338868abdfb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:15.061952 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6099af0fd3564a7f979d7235addd336d consensus state has no leader
I20250625 13:52:15.062083 1170 tablet_bootstrap.cc:654] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:15.062492 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c4a6d7d0c9647a4a17dca28d6a063a5 consensus state has no leader
I20250625 13:52:15.063832 2338 raft_consensus.cc:491] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.064034 2338 raft_consensus.cc:513] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.064450 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_98ed87f3ecee4add88ff785fe7175ce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.065018 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98ed87f3ecee4add88ff785fe7175ce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.065166 2338 leader_election.cc:290] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.065335 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:15.065794 849 leader_election.cc:343] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.069826 748 tablet_service.cc:1468] Processing CreateTablet for tablet a9969b2f1893446f9cec8338868abdfb (DEFAULT_TABLE table=default.Table_41a10889a8c045adb2961b8b2cde45c6 [id=c4fca0b2424b46c297de5330d494f5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.070786 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_50aa6fee302644afa6b847fd604555b3" } new_table_name: "default.Table_a029ff3e9b1448d2a8187d7e9cc8c170" modify_external_catalogs: true
I20250625 13:52:15.072335 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6bd08e23382b4c0880724d8266377783" } new_table_name: "default.Table_25b2b6109cdc4e76b9ef84637b24d955" modify_external_catalogs: true
I20250625 13:52:15.072700 891 tablet_service.cc:1468] Processing CreateTablet for tablet a9969b2f1893446f9cec8338868abdfb (DEFAULT_TABLE table=default.Table_41a10889a8c045adb2961b8b2cde45c6 [id=c4fca0b2424b46c297de5330d494f5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.073105 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9969b2f1893446f9cec8338868abdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.074414 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1bf18edd3d52428f9c11971f9df6b8a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.074901 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bf18edd3d52428f9c11971f9df6b8a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.075507 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a5675aff497c4f018385318a042343f4" } modify_external_catalogs: true
I20250625 13:52:15.075639 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a5675aff497c4f018385318a042343f4" } modify_external_catalogs: true
I20250625 13:52:15.080123 745 tablet_service.cc:1468] Processing CreateTablet for tablet 23fa1b7e7c13444fb8ee07807dabd813 (DEFAULT_TABLE table=default.Table_2b9e44578bd84aa6a63b1371d17d2318 [id=88935d7cbb94413bb33d6e79b37822e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.080325 744 tablet_service.cc:1468] Processing CreateTablet for tablet fec9b9d1133b457ea7901844629de683 (DEFAULT_TABLE table=default.Table_f85b46b8fe3f4f589471d8c43630f7ff [id=acd5080f18df4af38373493fb8a3a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.081082 1028 tablet_service.cc:1468] Processing CreateTablet for tablet fec9b9d1133b457ea7901844629de683 (DEFAULT_TABLE table=default.Table_f85b46b8fe3f4f589471d8c43630f7ff [id=acd5080f18df4af38373493fb8a3a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.081470 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec9b9d1133b457ea7901844629de683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.082635 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6e2d109e6fce407dad72a2469479b5ed
I20250625 13:52:15.084197 2338 raft_consensus.cc:491] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.084312 2338 raft_consensus.cc:513] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.084674 589 catalog_manager.cc:5582] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:15.084992 2338 leader_election.cc:290] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.085028 589 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:15.085199 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3859f399a7b84de99658380bd6c24ff5" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:15.085631 849 leader_election.cc:343] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.086056 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e2d109e6fce407dad72a2469479b5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.089917 749 tablet_service.cc:1468] Processing CreateTablet for tablet ce8010a588304411aedc2024c7a1ac77 (DEFAULT_TABLE table=default.Table_8d38330d3fb546dfb11c9ef7f6359da0 [id=840be0b31e23424d856171109e448b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.096726 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:15.096976 775 raft_consensus.cc:2466] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:15.097918 851 leader_election.cc:343] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3859f399a7b84de99658380bd6c24ff5
I20250625 13:52:15.098042 851 leader_election.cc:304] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.098156 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fa1b7e7c13444fb8ee07807dabd813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.098361 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e2d109e6fce407dad72a2469479b5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.098446 851 leader_election.cc:304] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:15.098649 2271 raft_consensus.cc:2747] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.098821 2271 raft_consensus.cc:2802] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:15.098881 2271 raft_consensus.cc:491] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:15.098939 2271 raft_consensus.cc:3058] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:15.099524 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9969b2f1893446f9cec8338868abdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.100778 886 tablet_service.cc:1468] Processing CreateTablet for tablet ce8010a588304411aedc2024c7a1ac77 (DEFAULT_TABLE table=default.Table_8d38330d3fb546dfb11c9ef7f6359da0 [id=840be0b31e23424d856171109e448b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.101140 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8010a588304411aedc2024c7a1ac77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.101773 2271 raft_consensus.cc:513] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.102322 2271 leader_election.cc:290] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.102526 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e2d109e6fce407dad72a2469479b5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.103233 1020 tablet_service.cc:1468] Processing CreateTablet for tablet ce8010a588304411aedc2024c7a1ac77 (DEFAULT_TABLE table=default.Table_8d38330d3fb546dfb11c9ef7f6359da0 [id=840be0b31e23424d856171109e448b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.103503 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec9b9d1133b457ea7901844629de683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.103626 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8010a588304411aedc2024c7a1ac77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.106832 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:15.107358 849 leader_election.cc:343] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.107565 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce8010a588304411aedc2024c7a1ac77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.108645 750 tablet_service.cc:1468] Processing CreateTablet for tablet 0c49eab755dd4f1cb16da4e10a24eecb (DEFAULT_TABLE table=default.Table_88fb3c8cc4d74fa0b7cbfcc1a4c97179 [id=617f5dfcd64d408ea32281faf0fa6739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.109278 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c49eab755dd4f1cb16da4e10a24eecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.111383 891 tablet_service.cc:1468] Processing CreateTablet for tablet fec9b9d1133b457ea7901844629de683 (DEFAULT_TABLE table=default.Table_f85b46b8fe3f4f589471d8c43630f7ff [id=acd5080f18df4af38373493fb8a3a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.111452 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:15.111609 775 raft_consensus.cc:3058] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:15.111760 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec9b9d1133b457ea7901844629de683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.114809 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7700dabe9c4446908a9535b1b946a3e0" } modify_external_catalogs: true
I20250625 13:52:15.124715 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 23fa1b7e7c13444fb8ee07807dabd813 (DEFAULT_TABLE table=default.Table_2b9e44578bd84aa6a63b1371d17d2318 [id=88935d7cbb94413bb33d6e79b37822e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.125130 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fa1b7e7c13444fb8ee07807dabd813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.125964 882 tablet_service.cc:1468] Processing CreateTablet for tablet 23fa1b7e7c13444fb8ee07807dabd813 (DEFAULT_TABLE table=default.Table_2b9e44578bd84aa6a63b1371d17d2318 [id=88935d7cbb94413bb33d6e79b37822e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.126354 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23fa1b7e7c13444fb8ee07807dabd813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.126368 1166 tablet_bootstrap.cc:492] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.126543 1166 ts_tablet_manager.cc:1397] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.173s user 0.000s sys 0.003s
I20250625 13:52:15.132120 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:15.143210 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.143873 2348 consensus_queue.cc:1035] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:15.156888 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fec914ebb830498da551091c86a181cd" } modify_external_catalogs: true
I20250625 13:52:15.157222 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fec914ebb830498da551091c86a181cd" } modify_external_catalogs: true
W20250625 13:52:15.159045 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e4fbede3f9040709ae4bdc9bbb792ad consensus state has no leader
I20250625 13:52:15.159518 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bc5ae60b26494a4aa4ca38cfdb824db3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.160180 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc5ae60b26494a4aa4ca38cfdb824db3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:15.161972 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b610cb974e8c45eeb756278c193c5357 consensus state has no leader
I20250625 13:52:15.162385 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_feec880c1f294ed2a18e041a5bc7a405" } modify_external_catalogs: true
I20250625 13:52:15.162597 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_feec880c1f294ed2a18e041a5bc7a405" } modify_external_catalogs: true
I20250625 13:52:15.164528 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1480ea8e34de4a38b7f925b0079ac328" } new_table_name: "default.Table_7261f19a2a1c44c8b14050450e5b8ce3" modify_external_catalogs: true
I20250625 13:52:15.167335 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e0e0b8f7f26d45d4af21d3f2c9219fa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.167980 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e0b8f7f26d45d4af21d3f2c9219fa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.169654 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7700dabe9c4446908a9535b1b946a3e0" } modify_external_catalogs: true
I20250625 13:52:15.179353 1168 ts_tablet_manager.cc:1939] T ac4d368347444846880954ced9651404 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.179483 1171 tablet_bootstrap.cc:654] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.179855 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 912) took 5091 ms. Trace:
I20250625 13:52:15.179941 1168 rpcz_store.cc:276] 0625 13:52:10.088459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.088503 (+ 44us) service_pool.cc:225] Handling call
0625 13:52:14.804466 (+4715963us) ts_tablet_manager.cc:1145] Deleting tablet ac4d368347444846880954ced9651404
0625 13:52:14.804476 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.964865 (+160389us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:14.967952 (+ 3087us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.179835 (+211883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4715713}
I20250625 13:52:15.180433 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet ac4d368347444846880954ced9651404 (table default.Table_581fcddbe9bb428d84556e6156637b4a [id=3a7ba6e0a96c4f3fb656f9ac4f6ee82d]) successfully deleted
I20250625 13:52:15.197921 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 17cf1172bcc74cca930824f9cda95f3b
I20250625 13:52:15.199433 1171 tablet_bootstrap.cc:492] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:15.199824 1171 ts_tablet_manager.cc:1397] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.169s user 0.010s sys 0.002s
I20250625 13:52:15.201953 1171 raft_consensus.cc:357] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.202273 1171 raft_consensus.cc:383] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.202409 1171 raft_consensus.cc:738] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.203848 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 17cf1172bcc74cca930824f9cda95f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.204113 1168 tablet_replica.cc:331] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.204157 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 17cf1172bcc74cca930824f9cda95f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.203774 1171 consensus_queue.cc:260] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.205709 1171 ts_tablet_manager.cc:1428] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:52:15.206213 1171 tablet_bootstrap.cc:492] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:15.208767 1171 tablet_bootstrap.cc:654] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.215903 775 raft_consensus.cc:2466] T b54183cf36b049c7bed30deb1604c59a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:15.224211 1168 raft_consensus.cc:2241] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.224663 1168 pending_rounds.cc:70] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:15.224786 1168 pending_rounds.cc:77] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520651692707840 op_type: NO_OP noop_request { }
I20250625 13:52:15.224967 1168 raft_consensus.cc:2887] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:15.225123 1168 raft_consensus.cc:2270] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.229256 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.235556 894 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e with delete_type TABLET_DATA_DELETED (replaced by tablet 066db01f791542a38cfb6e408c4836ed) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.236487 851 leader_election.cc:304] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:15.236879 2338 raft_consensus.cc:2802] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:15.236989 2338 raft_consensus.cc:695] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:15.237202 2338 consensus_queue.cc:237] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.237435 1166 raft_consensus.cc:357] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.237586 1166 raft_consensus.cc:383] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.237634 1166 raft_consensus.cc:738] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.237810 1166 consensus_queue.cc:260] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.238270 1166 ts_tablet_manager.cc:1428] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.112s user 0.001s sys 0.002s
I20250625 13:52:15.238583 1166 tablet_bootstrap.cc:492] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:15.239737 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 17cf1172bcc74cca930824f9cda95f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.243083 1165 ts_tablet_manager.cc:1918] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250625 13:52:15.244657 1166 tablet_bootstrap.cc:654] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.248749 1168 ts_tablet_manager.cc:1905] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.248835 1165 log.cc:1199] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/3f56a8a6cb964d3a979e2f1d944eeddc
I20250625 13:52:15.258620 1165 ts_tablet_manager.cc:1939] T 3f56a8a6cb964d3a979e2f1d944eeddc P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:15.260145 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_2c311e79aef2487d8524720a4cd0677c [id=71729b0c8f7046bb8e80bb96f9f2ba71]) successfully deleted
I20250625 13:52:15.262843 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f47980e0c9ad487b8113f13458e175ec
I20250625 13:52:15.263906 894 tablet_service.cc:1515] Processing DeleteTablet for tablet f47980e0c9ad487b8113f13458e175ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.265326 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_843c4743b1b346f9a3fa1343436191ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.265865 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_843c4743b1b346f9a3fa1343436191ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.267331 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4d2946a0f06e4f3fa1cde181e94856f3" } new_table_name: "default.Table_bae4ae0d1a2d4285be128a582827a743" modify_external_catalogs: true
I20250625 13:52:15.267634 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_13ab9b8203fe4404afbd33bdd16d06e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.268018 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13ab9b8203fe4404afbd33bdd16d06e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.268450 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_521bb02a0fa64038b747db4f73fa8c90" } new_table_name: "default.Table_1306b81b1e1c4f1ca615ad9b7a878925" modify_external_catalogs: true
I20250625 13:52:15.269909 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_df9f76474cdb40fdb91c03605ebdee8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.270332 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df9f76474cdb40fdb91c03605ebdee8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.271680 758 tablet_service.cc:1515] Processing DeleteTablet for tablet f47980e0c9ad487b8113f13458e175ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.275113 2271 raft_consensus.cc:491] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.275240 2271 raft_consensus.cc:513] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.275796 2271 leader_election.cc:290] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.276358 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74884bf313724d678d3b5f5869f09f97" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:15.276882 849 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.280301 1170 tablet_bootstrap.cc:492] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:15.281172 1170 ts_tablet_manager.cc:1397] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.232s user 0.027s sys 0.034s
I20250625 13:52:15.282064 2254 rpcz_store.cc:276] 0625 13:52:12.997809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.997983 (+ 174us) service_pool.cc:225] Handling call
0625 13:52:14.997306 (+1999323us) ts_tablet_manager.cc:1145] Deleting tablet 909f4bb91e784b29acbbc9763e35ab03
0625 13:52:14.997313 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:14.997392 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999160}
W20250625 13:52:15.283177 851 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 74884bf313724d678d3b5f5869f09f97
I20250625 13:52:15.283308 851 leader_election.cc:304] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.283845 2210 raft_consensus.cc:2747] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.283845 2271 raft_consensus.cc:491] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.284001 2271 raft_consensus.cc:513] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.284525 2271 leader_election.cc:290] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:15.285820 851 leader_election.cc:343] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 63d01af6b320489fb99c467c11ed1dc0
I20250625 13:52:15.307981 1168 ts_tablet_manager.cc:1918] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:15.310199 1168 log.cc:1199] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/df1ed922750f45eab02d1fc0c26af75e
I20250625 13:52:15.310884 1168 ts_tablet_manager.cc:1939] T df1ed922750f45eab02d1fc0c26af75e P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.311365 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9252dde349d24a8c99dfe2eda37c9f64
I20250625 13:52:15.312443 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 914) took 5223 ms. Trace:
I20250625 13:52:15.313171 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 9252dde349d24a8c99dfe2eda37c9f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.313995 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 9252dde349d24a8c99dfe2eda37c9f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:15.314684 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7070fcfd40e40bfb1908b4c4013c842 consensus state has no leader
I20250625 13:52:15.317234 1166 tablet_bootstrap.cc:492] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.317425 1166 ts_tablet_manager.cc:1397] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.000s
I20250625 13:52:15.318312 1166 raft_consensus.cc:357] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.318547 1166 raft_consensus.cc:383] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.318631 1166 raft_consensus.cc:738] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.318884 1166 consensus_queue.cc:260] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.319484 1166 ts_tablet_manager.cc:1428] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:15.312588 1168 rpcz_store.cc:276] 0625 13:52:10.088732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.088794 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:15.204026 (+5115232us) ts_tablet_manager.cc:1145] Deleting tablet df1ed922750f45eab02d1fc0c26af75e
0625 13:52:15.204037 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.224581 (+ 20544us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x6160001e1c54. stack: 00007f37024dc336 00007f37024dc0f0 00007f37014d23ad 0000000000971004 0000000000970fcd 000000000097099a 00007f370df3c000 00007f370f24e093 00007f3711d2b137 00007f3711d5d1c4 00007f3711d3ec20 00007f3711d3e9ad 0000000000a7573c 00007f37025219df 00007f3702524cab 00007f3702524b28
0625 13:52:15.287597 (+ 63016us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.307842 (+ 20245us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.312416 (+ 4574us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45773696,"tablet-delete.queue_time_us":5115097}
I20250625 13:52:15.319898 1166 tablet_bootstrap.cc:492] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:15.320281 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet df1ed922750f45eab02d1fc0c26af75e (table default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]) successfully deleted
I20250625 13:52:15.320734 1168 tablet_replica.cc:331] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.321173 1168 raft_consensus.cc:2241] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:15.321278 1166 tablet_bootstrap.cc:654] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.321765 1168 raft_consensus.cc:2270] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.322701 1168 ts_tablet_manager.cc:1905] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:15.323407 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bb9fcf91fa304a359213a3e5267a9cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb9fcf91fa304a359213a3e5267a9cac already in progress: creating tablet
W20250625 13:52:15.324075 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 83699efa521a44f092cb738002aa58e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83699efa521a44f092cb738002aa58e4 already in progress: creating tablet
W20250625 13:52:15.329458 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0346ed4f8d3f46aa84cbabbcadcebf23 already in progress: creating tablet
W20250625 13:52:15.330441 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9710609da88d4b42993b3aef02416543 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9710609da88d4b42993b3aef02416543 already in progress: creating tablet
W20250625 13:52:15.340035 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bbd3200bd1384187978aeb7089ec4e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbd3200bd1384187978aeb7089ec4e03 already in progress: creating tablet
W20250625 13:52:15.340693 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d2e8f4e3b45a4065b74485157f31247d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2e8f4e3b45a4065b74485157f31247d already in progress: creating tablet
W20250625 13:52:15.341245 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 29e223675c7842b383865a270b97ce58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29e223675c7842b383865a270b97ce58 already in progress: creating tablet
W20250625 13:52:15.341768 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 already in progress: creating tablet
I20250625 13:52:15.345814 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_78d7cfce8bd9428c8b6f47e48fc1ddb6" } modify_external_catalogs: true
I20250625 13:52:15.345970 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_78d7cfce8bd9428c8b6f47e48fc1ddb6" } modify_external_catalogs: true
I20250625 13:52:15.323422 1171 tablet_bootstrap.cc:492] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:15.342715 1168 ts_tablet_manager.cc:1918] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:15.347694 1168 log.cc:1199] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/0b129117db064a15b27e82439637cf22
I20250625 13:52:15.347817 1171 ts_tablet_manager.cc:1397] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.142s user 0.007s sys 0.000s
I20250625 13:52:15.348105 1168 ts_tablet_manager.cc:1939] T 0b129117db064a15b27e82439637cf22 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.348919 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 911) took 5260 ms. Trace:
I20250625 13:52:15.349022 1168 rpcz_store.cc:276] 0625 13:52:10.088337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.089039 (+ 702us) service_pool.cc:225] Handling call
0625 13:52:15.320678 (+5231639us) ts_tablet_manager.cc:1145] Deleting tablet 0b129117db064a15b27e82439637cf22
0625 13:52:15.320688 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.330014 (+ 9326us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.342592 (+ 12578us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.348895 (+ 6303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5231420}
I20250625 13:52:15.349457 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 0b129117db064a15b27e82439637cf22 (table default.Table_78be58e64e564e6ab48ccee86fbe5279 [id=d3e9764fbbc14f7db3ea0dbf1ee02697]) successfully deleted
I20250625 13:52:15.349903 1168 tablet_replica.cc:331] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.350178 1168 raft_consensus.cc:2241] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:15.350535 1168 pending_rounds.cc:70] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:15.350626 1168 pending_rounds.cc:77] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520644097355776 op_type: NO_OP noop_request { }
I20250625 13:52:15.351272 1168 raft_consensus.cc:2887] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:15.351738 1168 raft_consensus.cc:2270] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.352435 1168 ts_tablet_manager.cc:1905] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:15.353892 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 0bc136dc84de44b49abcdf4100a86137 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bc136dc84de44b49abcdf4100a86137 already in progress: creating tablet
I20250625 13:52:15.355252 1171 raft_consensus.cc:357] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.355577 1171 raft_consensus.cc:383] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.357136 1171 raft_consensus.cc:738] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.359601 1168 ts_tablet_manager.cc:1918] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250625 13:52:15.360451 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6e2d109e6fce407dad72a2469479b5ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2d109e6fce407dad72a2469479b5ed already in progress: creating tablet
I20250625 13:52:15.360872 1165 tablet_replica.cc:331] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:15.361294 1165 raft_consensus.cc:2241] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.361809 1165 raft_consensus.cc:2270] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.362387 1165 ts_tablet_manager.cc:1905] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.357586 1171 consensus_queue.cc:260] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.365531 1171 ts_tablet_manager.cc:1428] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 13:52:15.366449 2419 tablet.cc:1722] T 20c12f55ed5b4425aea1bf45cdc0658e P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:15.370008 590 master_service.cc:948] ReplaceTablet: received request to replace tablet c737daaede054d09a0c07564157423a9 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:15.367043 2420 tablet.cc:1722] T 20c12f55ed5b4425aea1bf45cdc0658e P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:15.370838 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_12d22cb86cd5435c84a242d6e0eb12de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.372406 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12d22cb86cd5435c84a242d6e0eb12de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.372082 1171 tablet_bootstrap.cc:492] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:15.364804 1168 log.cc:1199] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/77c50e1ebb9042ef8f0a651ba7186b74
I20250625 13:52:15.375630 1168 ts_tablet_manager.cc:1939] T 77c50e1ebb9042ef8f0a651ba7186b74 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.376394 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 916) took 5287 ms. Trace:
I20250625 13:52:15.376545 1168 rpcz_store.cc:276] 0625 13:52:10.088998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.089039 (+ 41us) service_pool.cc:225] Handling call
0625 13:52:15.349854 (+5260815us) ts_tablet_manager.cc:1145] Deleting tablet 77c50e1ebb9042ef8f0a651ba7186b74
0625 13:52:15.349865 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.356687 (+ 6822us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.359518 (+ 2831us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.376375 (+ 16857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5260440}
I20250625 13:52:15.377009 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 77c50e1ebb9042ef8f0a651ba7186b74 (table default.Table_bdcf17e5612a44579ba27af546ca1def [id=cc778fd879634dd6a2ba7ba8274c9ef4]) successfully deleted
I20250625 13:52:15.377314 1168 tablet_replica.cc:331] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.378666 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d0baddb123f0401a98662b4b483b9587
I20250625 13:52:15.379743 1168 raft_consensus.cc:2241] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.379874 1168 raft_consensus.cc:2270] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.380295 1168 ts_tablet_manager.cc:1905] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.381124 2338 raft_consensus.cc:491] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.381234 2338 raft_consensus.cc:513] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.382721 2338 leader_election.cc:290] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.383926 882 tablet_service.cc:1515] Processing DeleteTablet for tablet d0baddb123f0401a98662b4b483b9587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:15.386116 851 leader_election.cc:343] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cbe4046742b34f1ab0b1610b5d6ee01e
I20250625 13:52:15.392369 750 tablet_service.cc:1515] Processing DeleteTablet for tablet d0baddb123f0401a98662b4b483b9587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.285022 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63d01af6b320489fb99c467c11ed1dc0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.289819 2414 raft_consensus.cc:491] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.395543 2414 raft_consensus.cc:513] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.395941 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 909f4bb91e784b29acbbc9763e35ab03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 909f4bb91e784b29acbbc9763e35ab03 already in progress: creating tablet
I20250625 13:52:15.396171 2414 leader_election.cc:290] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:15.301237 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet f47980e0c9ad487b8113f13458e175ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.396310 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1128) took 2398 ms. Trace:
W20250625 13:52:15.301932 983 consensus_peers.cc:487] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:15.396382 2254 rpcz_store.cc:276] 0625 13:52:12.997931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.998296 (+ 365us) service_pool.cc:225] Handling call
0625 13:52:15.396203 (+2397907us) ts_tablet_manager.cc:1145] Deleting tablet 9df1f195befe4aed999b41f770547fde
0625 13:52:15.396215 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.396297 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397735}
W20250625 13:52:15.396499 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 95052.9us
I20250625 13:52:15.396440 1171 tablet_bootstrap.cc:654] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:15.396742 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9df1f195befe4aed999b41f770547fde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9df1f195befe4aed999b41f770547fde already in progress: creating tablet
W20250625 13:52:15.396941 849 leader_election.cc:343] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.397056 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1131) took 2397 ms. Trace:
I20250625 13:52:15.397042 849 leader_election.cc:304] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.397116 2254 rpcz_store.cc:276] 0625 13:52:12.999290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.999356 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:15.396947 (+2397591us) ts_tablet_manager.cc:1145] Deleting tablet dfaba56442c0498da95e6fad126ab86d
0625 13:52:15.396955 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.397024 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397419}
I20250625 13:52:15.300242 2410 raft_consensus.cc:491] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:15.397419 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet dfaba56442c0498da95e6fad126ab86d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfaba56442c0498da95e6fad126ab86d already in progress: creating tablet
I20250625 13:52:15.299206 1170 raft_consensus.cc:357] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.315657 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 9252dde349d24a8c99dfe2eda37c9f64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.397432 2410 raft_consensus.cc:513] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.397583 1170 raft_consensus.cc:383] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.397644 1170 raft_consensus.cc:738] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.397708 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1130) took 2398 ms. Trace:
I20250625 13:52:15.397770 2254 rpcz_store.cc:276] 0625 13:52:12.999197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.999559 (+ 362us) service_pool.cc:225] Handling call
0625 13:52:15.397619 (+2398060us) ts_tablet_manager.cc:1145] Deleting tablet fd7a6587aa674a0eaaa107e2a2f673a3
0625 13:52:15.397628 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.397694 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397937}
I20250625 13:52:15.397826 1170 consensus_queue.cc:260] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.397974 2410 leader_election.cc:290] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:15.398087 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet fd7a6587aa674a0eaaa107e2a2f673a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd7a6587aa674a0eaaa107e2a2f673a3 already in progress: creating tablet
I20250625 13:52:15.398294 1170 ts_tablet_manager.cc:1428] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.117s user 0.000s sys 0.002s
W20250625 13:52:15.325749 985 consensus_peers.cc:487] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df1ed922750f45eab02d1fc0c26af75e. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:15.398456 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 72761.5us
I20250625 13:52:15.398499 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "430c3bed360b4f32855a8adb99076608" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:15.398655 915 raft_consensus.cc:2466] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:15.382089 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe4046742b34f1ab0b1610b5d6ee01e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.384359 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet d0baddb123f0401a98662b4b483b9587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:15.400542 985 leader_election.cc:343] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 430c3bed360b4f32855a8adb99076608
W20250625 13:52:15.400765 849 leader_election.cc:343] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.400856 849 leader_election.cc:304] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.401295 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_190dea7daa454753b901b65cd213f592"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.401832 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_190dea7daa454753b901b65cd213f592 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.401839 2271 raft_consensus.cc:2747] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.401999 2271 raft_consensus.cc:2747] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.402191 1170 tablet_bootstrap.cc:492] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:15.402664 983 leader_election.cc:304] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.402861 2411 consensus_queue.cc:1035] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:15.403203 2411 raft_consensus.cc:2802] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:15.403266 2411 raft_consensus.cc:491] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:15.403359 2411 raft_consensus.cc:3058] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:15.403885 985 leader_election.cc:343] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 024f6cb363594420b0d5480fa79d156b
I20250625 13:52:15.406019 2411 raft_consensus.cc:513] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.406272 985 consensus_peers.cc:487] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b14d00c93a964edd815122e3723f89e7. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:15.406737 2411 leader_election.cc:290] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:15.407680 983 leader_election.cc:343] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 024f6cb363594420b0d5480fa79d156b
I20250625 13:52:15.407766 983 leader_election.cc:304] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:15.408110 983 consensus_peers.cc:487] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:15.408697 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "430c3bed360b4f32855a8adb99076608" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:15.408905 913 raft_consensus.cc:3058] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:15.410202 1170 tablet_bootstrap.cc:654] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.410553 2411 raft_consensus.cc:2747] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:15.412551 985 leader_election.cc:343] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 430c3bed360b4f32855a8adb99076608
I20250625 13:52:15.413955 1168 ts_tablet_manager.cc:1918] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:15.420238 1166 tablet_bootstrap.cc:492] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.420451 1166 ts_tablet_manager.cc:1397] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.000s
I20250625 13:52:15.421638 1166 raft_consensus.cc:357] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.421890 1166 raft_consensus.cc:383] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.421983 1166 raft_consensus.cc:738] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.422243 1166 consensus_queue.cc:260] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.422885 1166 ts_tablet_manager.cc:1428] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:15.423171 1166 tablet_bootstrap.cc:492] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:15.424839 1166 tablet_bootstrap.cc:654] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.424865 1168 log.cc:1199] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/b14d00c93a964edd815122e3723f89e7
I20250625 13:52:15.425343 1168 ts_tablet_manager.cc:1939] T b14d00c93a964edd815122e3723f89e7 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.425388 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:15.425761 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 913) took 5337 ms. Trace:
I20250625 13:52:15.425884 1168 rpcz_store.cc:276] 0625 13:52:10.088605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.089311 (+ 706us) service_pool.cc:225] Handling call
0625 13:52:15.377267 (+5287956us) ts_tablet_manager.cc:1145] Deleting tablet b14d00c93a964edd815122e3723f89e7
0625 13:52:15.377276 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.389104 (+ 11828us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.413848 (+ 24744us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.425744 (+ 11896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5287780}
I20250625 13:52:15.426399 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet b14d00c93a964edd815122e3723f89e7 (table default.Table_295dd4f121234e659090b8e4cf70faee [id=47cd45886fc94cd7ba5cafa60838d601]) successfully deleted
I20250625 13:52:15.426772 1168 tablet_replica.cc:331] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.426968 1168 raft_consensus.cc:2241] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.427148 1168 raft_consensus.cc:2270] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.427731 1168 ts_tablet_manager.cc:1905] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.414711 913 raft_consensus.cc:2466] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:15.430750 983 leader_election.cc:304] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.431185 2411 raft_consensus.cc:2802] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:15.431293 2411 raft_consensus.cc:695] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:15.431582 2411 consensus_queue.cc:237] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.440197 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8684eb9104e44227944b21784c54e496 consensus state has no leader
I20250625 13:52:15.440550 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b1ac14afc057412d92fba0065ebb7a1b" } new_table_name: "default.Table_a6c0a9a6d69f4cb0aa50ddb77f00a141" modify_external_catalogs: true
I20250625 13:52:15.441426 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_85881e844bd44e0e97ff03992f7359c1" } new_table_name: "default.Table_6c11e5622867415387580cfacdb821aa" modify_external_catalogs: true
W20250625 13:52:15.441816 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 234be8f208324ddd969555a2e88e2546 consensus state has no leader
I20250625 13:52:15.442101 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a4622e5472934b05b366bfa81a23ddf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.442581 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4622e5472934b05b366bfa81a23ddf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.443364 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 490c8902efeb46c99543747a8787c1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.445009 1168 ts_tablet_manager.cc:1918] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:15.445482 2338 raft_consensus.cc:491] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.447193 2338 raft_consensus.cc:513] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.447964 1168 log.cc:1199] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/4fa801f7d98e4794b6ca439638f29e69
I20250625 13:52:15.450994 2338 leader_election.cc:290] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.452313 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_01b8d89dbb714bdc83d42185cc4a1fd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.452852 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01b8d89dbb714bdc83d42185cc4a1fd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.455022 1168 ts_tablet_manager.cc:1939] T 4fa801f7d98e4794b6ca439638f29e69 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.455453 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 915) took 5366 ms. Trace:
I20250625 13:52:15.455588 1168 rpcz_store.cc:276] 0625 13:52:10.088871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.089108 (+ 237us) service_pool.cc:225] Handling call
0625 13:52:15.426724 (+5337616us) ts_tablet_manager.cc:1145] Deleting tablet 4fa801f7d98e4794b6ca439638f29e69
0625 13:52:15.426736 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.439398 (+ 12662us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.444907 (+ 5509us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.455437 (+ 10530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5337182}
I20250625 13:52:15.456090 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 4fa801f7d98e4794b6ca439638f29e69 (table default.Table_e1f377cdd7b74dfaaf9b2d7d3af42328 [id=1e5117fcc9bb41c3bb98021f499b766c]) successfully deleted
I20250625 13:52:15.456429 1168 tablet_replica.cc:331] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:15.460084 851 leader_election.cc:343] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 70959ec681524518a7ec2c9ed4016ad1
I20250625 13:52:15.461835 1168 raft_consensus.cc:2241] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.462070 1168 raft_consensus.cc:2270] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.462687 1168 ts_tablet_manager.cc:1905] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.467994 1171 tablet_bootstrap.cc:492] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:15.468173 1171 ts_tablet_manager.cc:1397] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.000s
I20250625 13:52:15.468952 1171 raft_consensus.cc:357] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.471383 1171 raft_consensus.cc:383] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.471510 1171 raft_consensus.cc:738] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.471815 1171 consensus_queue.cc:260] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.472504 1171 ts_tablet_manager.cc:1428] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:15.470203 1168 ts_tablet_manager.cc:1918] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:15.472823 1171 tablet_bootstrap.cc:492] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:15.473934 1168 log.cc:1199] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/430c3bed360b4f32855a8adb99076608
I20250625 13:52:15.474689 1168 ts_tablet_manager.cc:1939] T 430c3bed360b4f32855a8adb99076608 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.475641 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 917) took 5346 ms. Trace:
I20250625 13:52:15.475775 1168 rpcz_store.cc:276] 0625 13:52:10.128992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.129055 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:15.456383 (+5327328us) ts_tablet_manager.cc:1145] Deleting tablet 430c3bed360b4f32855a8adb99076608
0625 13:52:15.456393 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.467099 (+ 10706us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.470123 (+ 3024us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.475622 (+ 5499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5327044}
I20250625 13:52:15.476006 1165 ts_tablet_manager.cc:1918] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:15.477051 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 430c3bed360b4f32855a8adb99076608 (table default.Table_1099b7a2614540d3ac476e1bad30554c [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]) successfully deleted
I20250625 13:52:15.477043 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70959ec681524518a7ec2c9ed4016ad1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.477442 1168 tablet_replica.cc:331] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:15.477582 849 leader_election.cc:343] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.477681 849 leader_election.cc:304] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.478042 2338 raft_consensus.cc:2747] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.477675 1168 raft_consensus.cc:2241] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.478435 1168 raft_consensus.cc:2270] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.479020 1168 ts_tablet_manager.cc:1905] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.478546 1171 tablet_bootstrap.cc:654] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.489342 1168 ts_tablet_manager.cc:1918] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:15.490188 1168 log.cc:1199] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/2404e712e18146b68757bd77362c1358
I20250625 13:52:15.490711 1168 ts_tablet_manager.cc:1939] T 2404e712e18146b68757bd77362c1358 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.491472 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 918) took 5361 ms. Trace:
I20250625 13:52:15.492053 1168 rpcz_store.cc:276] 0625 13:52:10.129797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.129949 (+ 152us) service_pool.cc:225] Handling call
0625 13:52:15.477396 (+5347447us) ts_tablet_manager.cc:1145] Deleting tablet 2404e712e18146b68757bd77362c1358
0625 13:52:15.477406 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.483034 (+ 5628us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.489234 (+ 6200us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.491454 (+ 2220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5347253}
I20250625 13:52:15.492908 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 2404e712e18146b68757bd77362c1358 (table default.Table_4dc8c488c3e44f3f882049d1c0f8feb2 [id=0532faedda28481f9742008aecb6bfd3]) successfully deleted
I20250625 13:52:15.493415 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 919) took 5310 ms. Trace:
I20250625 13:52:15.500031 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1129) took 2500 ms. Trace:
I20250625 13:52:15.500294 2254 rpcz_store.cc:276] 0625 13:52:12.999105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.999698 (+ 593us) service_pool.cc:225] Handling call
0625 13:52:15.400729 (+2401031us) ts_tablet_manager.cc:1145] Deleting tablet a1263945be894618a7f9946e7ca50efd
0625 13:52:15.476155 (+ 75426us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.499976 (+ 23821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2400910}
I20250625 13:52:15.501190 2254 tablet_replica.cc:331] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:15.501621 2254 raft_consensus.cc:2241] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:15.502261 2254 raft_consensus.cc:2270] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.504326 1170 tablet_bootstrap.cc:492] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:15.504640 1170 ts_tablet_manager.cc:1397] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.103s user 0.001s sys 0.008s
W20250625 13:52:15.505452 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a1263945be894618a7f9946e7ca50efd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1263945be894618a7f9946e7ca50efd already in progress: creating tablet
I20250625 13:52:15.506095 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_4408ed6957e948b08dee253e7781b84e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.506618 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4408ed6957e948b08dee253e7781b84e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.508792 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_312dbec5d3ab4810aeb370951ebfafde" } modify_external_catalogs: true
I20250625 13:52:15.508939 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_312dbec5d3ab4810aeb370951ebfafde" } modify_external_catalogs: true
I20250625 13:52:15.510427 1170 raft_consensus.cc:357] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.510587 1170 raft_consensus.cc:383] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.510632 1170 raft_consensus.cc:738] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.510919 1170 consensus_queue.cc:260] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.511555 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c55c56331834053a65654bb59304b61" } modify_external_catalogs: true
I20250625 13:52:15.511698 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7c55c56331834053a65654bb59304b61" } modify_external_catalogs: true
I20250625 13:52:15.511978 1170 ts_tablet_manager.cc:1428] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 13:52:15.478879 1166 tablet_bootstrap.cc:492] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.513726 1166 ts_tablet_manager.cc:1397] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.000s
I20250625 13:52:15.515074 1166 raft_consensus.cc:357] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.515249 1166 raft_consensus.cc:383] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.515295 1166 raft_consensus.cc:738] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:15.516721 985 consensus_peers.cc:487] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87f5b36ddb474ed6903c14876b236ba8. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:15.517280 985 consensus_peers.cc:487] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2404e712e18146b68757bd77362c1358. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:15.515503 1166 consensus_queue.cc:260] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.518857 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc0f3e2955344929b261f6affa378084 consensus state has no leader
W20250625 13:52:15.519704 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c00d7323dcae453683206e12ad22e1c8 consensus state has no leader
I20250625 13:52:15.519755 2254 ts_tablet_manager.cc:1905] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.521246 1166 ts_tablet_manager.cc:1428] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 13:52:15.521593 1166 tablet_bootstrap.cc:492] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:15.525208 1166 tablet_bootstrap.cc:654] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.527837 2338 raft_consensus.cc:491] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.527978 2338 raft_consensus.cc:513] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.528578 2338 leader_election.cc:290] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:15.529562 851 leader_election.cc:343] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7ba1338d0562418eadabdb8dbcaedee2
I20250625 13:52:15.529656 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ba1338d0562418eadabdb8dbcaedee2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.529847 1044 raft_consensus.cc:2466] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:15.530237 849 leader_election.cc:304] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.530835 2338 raft_consensus.cc:2802] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:15.530917 2338 raft_consensus.cc:491] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:15.530972 2338 raft_consensus.cc:3058] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:15.532768 1165 log.cc:1199] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/077e8c001cc646559b5268d4a48c0ad6
I20250625 13:52:15.533365 1165 ts_tablet_manager.cc:1939] T 077e8c001cc646559b5268d4a48c0ad6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:15.534566 2338 raft_consensus.cc:513] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.534842 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 077e8c001cc646559b5268d4a48c0ad6 (table default.Table_517319d59c234fe9a5fe83530d3a00c6 [id=56d2c709e389405c80e4fa20ce75d5ad]) successfully deleted
I20250625 13:52:15.535684 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ba1338d0562418eadabdb8dbcaedee2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:15.535856 1044 raft_consensus.cc:3058] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:15.535971 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 17cf1172bcc74cca930824f9cda95f3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17cf1172bcc74cca930824f9cda95f3b already in progress: creating tablet
W20250625 13:52:15.536598 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f47980e0c9ad487b8113f13458e175ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47980e0c9ad487b8113f13458e175ec already in progress: creating tablet
W20250625 13:52:15.537202 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 9252dde349d24a8c99dfe2eda37c9f64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9252dde349d24a8c99dfe2eda37c9f64 already in progress: creating tablet
W20250625 13:52:15.537801 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d0baddb123f0401a98662b4b483b9587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0baddb123f0401a98662b4b483b9587 already in progress: creating tablet
W20250625 13:52:15.538735 851 leader_election.cc:343] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7ba1338d0562418eadabdb8dbcaedee2
I20250625 13:52:15.538484 2338 leader_election.cc:290] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.493824 1168 rpcz_store.cc:276] 0625 13:52:10.182475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.182932 (+ 457us) service_pool.cc:225] Handling call
0625 13:52:15.493284 (+5310352us) ts_tablet_manager.cc:1145] Deleting tablet 0012e95ff4924723a6e6aaf6b85da438
0625 13:52:15.493294 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.493399 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5310105}
W20250625 13:52:15.541612 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 0012e95ff4924723a6e6aaf6b85da438 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0012e95ff4924723a6e6aaf6b85da438 already in progress: creating tablet
I20250625 13:52:15.541934 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap replaying log segment 1/1 (99.7K/134.6K this segment, stats: ops{read=635 overwritten=0 applied=321 ignored=0} inserts{seen=303 ignored=0} mutations{seen=367 ignored=0} orphaned_commits=0)
I20250625 13:52:15.542223 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 920) took 5349 ms. Trace:
I20250625 13:52:15.542313 1168 rpcz_store.cc:276] 0625 13:52:10.193178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.193242 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:15.542105 (+5348863us) ts_tablet_manager.cc:1145] Deleting tablet 156da2a965c64d1090a4f9d4eafdec70
0625 13:52:15.542117 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.542207 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5348605}
I20250625 13:52:15.542954 2338 raft_consensus.cc:491] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:15.543247 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 156da2a965c64d1090a4f9d4eafdec70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156da2a965c64d1090a4f9d4eafdec70 already in progress: creating tablet
I20250625 13:52:15.543220 2338 raft_consensus.cc:513] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.543713 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 922) took 5339 ms. Trace:
I20250625 13:52:15.543792 1168 rpcz_store.cc:276] 0625 13:52:10.204369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.204444 (+ 75us) service_pool.cc:225] Handling call
0625 13:52:15.543606 (+5339162us) ts_tablet_manager.cc:1145] Deleting tablet abc44a41b30e4a53977ec81d6905612f
0625 13:52:15.543615 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.543691 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5338857}
I20250625 13:52:15.543915 2338 leader_election.cc:290] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:15.544152 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet abc44a41b30e4a53977ec81d6905612f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abc44a41b30e4a53977ec81d6905612f already in progress: creating tablet
I20250625 13:52:15.544904 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 923) took 5328 ms. Trace:
I20250625 13:52:15.545228 1168 rpcz_store.cc:276] 0625 13:52:10.216748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.216799 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:15.544816 (+5328017us) ts_tablet_manager.cc:1145] Deleting tablet af934858a4af4621bb55eb5a528fa14c
0625 13:52:15.544824 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.544891 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5327821}
W20250625 13:52:15.545907 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet af934858a4af4621bb55eb5a528fa14c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af934858a4af4621bb55eb5a528fa14c already in progress: creating tablet
I20250625 13:52:15.546262 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 924) took 5328 ms. Trace:
I20250625 13:52:15.546571 1168 rpcz_store.cc:276] 0625 13:52:10.217484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.217544 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:15.546175 (+5328631us) ts_tablet_manager.cc:1145] Deleting tablet 3beb5dcb4f1e4ece821c83c3a52d93f5
0625 13:52:15.546183 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.546249 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5328435}
W20250625 13:52:15.547201 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3beb5dcb4f1e4ece821c83c3a52d93f5 already in progress: creating tablet
I20250625 13:52:15.547606 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 925) took 5329 ms. Trace:
I20250625 13:52:15.547936 1168 rpcz_store.cc:276] 0625 13:52:10.218077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.218125 (+ 48us) service_pool.cc:225] Handling call
0625 13:52:15.547517 (+5329392us) ts_tablet_manager.cc:1145] Deleting tablet d5cb3b35a5c2483fa82226133a7c161d
0625 13:52:15.547526 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.547591 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5329208}
W20250625 13:52:15.548527 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d5cb3b35a5c2483fa82226133a7c161d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5cb3b35a5c2483fa82226133a7c161d already in progress: creating tablet
W20250625 13:52:15.548990 985 consensus_peers.cc:487] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38a23b08a03b48eaa051f670eb7dac65. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:15.549047 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 959) took 4989 ms. Trace:
W20250625 13:52:15.549103 851 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: da2f8d0d9a714a28ba6fb9642d461de2
I20250625 13:52:15.549114 1168 rpcz_store.cc:276] 0625 13:52:10.559105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.605556 (+ 46451us) service_pool.cc:225] Handling call
0625 13:52:15.548936 (+4943380us) ts_tablet_manager.cc:1145] Deleting tablet 973c483c87b84ea09b6c7d556e54b490
0625 13:52:15.548952 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.549033 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4943143}
W20250625 13:52:15.549789 983 consensus_peers.cc:487] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2404e712e18146b68757bd77362c1358. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:15.549886 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 973c483c87b84ea09b6c7d556e54b490 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 973c483c87b84ea09b6c7d556e54b490 already in progress: creating tablet
I20250625 13:52:15.550370 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 960) took 4991 ms. Trace:
I20250625 13:52:15.550452 1168 rpcz_store.cc:276] 0625 13:52:10.559369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.605815 (+ 46446us) service_pool.cc:225] Handling call
0625 13:52:15.550271 (+4944456us) ts_tablet_manager.cc:1145] Deleting tablet a1a9a69d26c14a62b26e0e90157782af
0625 13:52:15.550281 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.550354 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4944299}
W20250625 13:52:15.550818 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a1a9a69d26c14a62b26e0e90157782af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a9a69d26c14a62b26e0e90157782af already in progress: creating tablet
I20250625 13:52:15.551169 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 961) took 4991 ms. Trace:
I20250625 13:52:15.551287 1168 rpcz_store.cc:276] 0625 13:52:10.559604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.605985 (+ 46381us) service_pool.cc:225] Handling call
0625 13:52:15.551073 (+4945088us) ts_tablet_manager.cc:1145] Deleting tablet e3939ef22ea344328721dfba67d933d1
0625 13:52:15.551080 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.551149 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4944949}
W20250625 13:52:15.551848 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e3939ef22ea344328721dfba67d933d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3939ef22ea344328721dfba67d933d1 already in progress: creating tablet
I20250625 13:52:15.552235 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 964) took 4991 ms. Trace:
I20250625 13:52:15.552372 1168 rpcz_store.cc:276] 0625 13:52:10.560674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.607672 (+ 46998us) service_pool.cc:225] Handling call
0625 13:52:15.552137 (+4944465us) ts_tablet_manager.cc:1145] Deleting tablet 47c70919181d4c5392f65df221ec6348
0625 13:52:15.552147 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.552220 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4944276}
W20250625 13:52:15.552865 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 47c70919181d4c5392f65df221ec6348 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c70919181d4c5392f65df221ec6348 already in progress: creating tablet
I20250625 13:52:15.553610 1171 tablet_bootstrap.cc:492] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:15.554054 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1e889135ee254afca2317cec6787455e" } new_table_name: "default.Table_b5709d271da1440b9ffa6a7289f660c2" modify_external_catalogs: true
I20250625 13:52:15.554275 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8af8a34363004cbaabfa6b6a8371425f" } modify_external_catalogs: true
I20250625 13:52:15.554421 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8af8a34363004cbaabfa6b6a8371425f" } modify_external_catalogs: true
I20250625 13:52:15.555297 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_491bf1ae35e74854b2d7f3f716215bbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.555814 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_491bf1ae35e74854b2d7f3f716215bbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.555905 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8f3b560e7437430287394d43472d24fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.556296 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f3b560e7437430287394d43472d24fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.553798 1171 ts_tablet_manager.cc:1397] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.008s
I20250625 13:52:15.557718 1171 raft_consensus.cc:357] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.557929 1171 raft_consensus.cc:383] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.558017 1171 raft_consensus.cc:738] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.558614 1166 tablet_bootstrap.cc:492] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.558808 1166 ts_tablet_manager.cc:1397] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20250625 13:52:15.558708 1171 consensus_queue.cc:260] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.559751 1166 raft_consensus.cc:357] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.559988 1166 raft_consensus.cc:383] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.560099 1166 raft_consensus.cc:738] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.560353 1166 consensus_queue.cc:260] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.560999 1166 ts_tablet_manager.cc:1428] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:15.564095 595 catalog_manager.cc:5582] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:15.565178 1171 ts_tablet_manager.cc:1428] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:52:15.565689 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 38770320fc6541c39e10448978f8a3b0 (table default.Table_6d9870087cdc4dfca8fa438020887d80 [id=43487ee990e640bcbeefc7c142075998]). Expected version 2 got 0
I20250625 13:52:15.569008 2414 raft_consensus.cc:491] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.569170 2414 raft_consensus.cc:513] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.569836 2414 leader_election.cc:290] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:15.571682 985 leader_election.cc:343] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:15.572899 750 tablet_service.cc:1468] Processing CreateTablet for tablet a0d4536a3a69448e9d9acca67393f0da (DEFAULT_TABLE table=default.Table_54c3accf07ed49259aa5cfdfcd715f72 [id=c9f77c0be7544bc7abd7269fb3fdaab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.573328 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d4536a3a69448e9d9acca67393f0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.573478 882 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3447f143704d9e835ff4f0dcff297b (DEFAULT_TABLE table=default.Table_cfe03d5988ab4502a2f04e1eef206474 [id=729d4515b71a4cd081492b5174e08635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.573848 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3447f143704d9e835ff4f0dcff297b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.576027 1166 tablet_bootstrap.cc:492] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:15.576323 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 38770320fc6541c39e10448978f8a3b0 (table default.Table_6d9870087cdc4dfca8fa438020887d80 [id=43487ee990e640bcbeefc7c142075998]). Expected version 2 got 0
I20250625 13:52:15.580387 894 tablet_service.cc:1468] Processing CreateTablet for tablet a0d4536a3a69448e9d9acca67393f0da (DEFAULT_TABLE table=default.Table_54c3accf07ed49259aa5cfdfcd715f72 [id=c9f77c0be7544bc7abd7269fb3fdaab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.580781 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d4536a3a69448e9d9acca67393f0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.581656 1166 tablet_bootstrap.cc:654] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:15.583468 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet b610cb974e8c45eeb756278c193c5357 (table default.Table_a029ff3e9b1448d2a8187d7e9cc8c170 [id=106eebd9055841afa03aa5f75c3b0e1d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b610cb974e8c45eeb756278c193c5357 consensus state has no leader
I20250625 13:52:15.585052 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3447f143704d9e835ff4f0dcff297b (DEFAULT_TABLE table=default.Table_cfe03d5988ab4502a2f04e1eef206474 [id=729d4515b71a4cd081492b5174e08635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.585413 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3447f143704d9e835ff4f0dcff297b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.588537 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4a226ca5c5c84ef380d6c37dd08f6740" } modify_external_catalogs: true
I20250625 13:52:15.588696 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_4a226ca5c5c84ef380d6c37dd08f6740" } modify_external_catalogs: true
I20250625 13:52:15.588729 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9068732806ae4ecd97e627db790b0a3d (DEFAULT_TABLE table=default.Table_a27119266aa84cf295b69911fabab0af [id=fb854116fef744de92549cab273c5195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.589116 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9068732806ae4ecd97e627db790b0a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.590245 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9068732806ae4ecd97e627db790b0a3d (DEFAULT_TABLE table=default.Table_a27119266aa84cf295b69911fabab0af [id=fb854116fef744de92549cab273c5195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.590608 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9068732806ae4ecd97e627db790b0a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.591089 890 tablet_service.cc:1468] Processing CreateTablet for tablet 99edfa13925641f3bf062b7daad5a395 (DEFAULT_TABLE table=default.Table_dec20faad45645eab2f7db58d65b3932 [id=e48fcb0bf7844fb18eb4562a6fd3dd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.591447 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99edfa13925641f3bf062b7daad5a395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.591606 748 tablet_service.cc:1468] Processing CreateTablet for tablet 24cd20660f4b48658775ddf14e6cf586 (DEFAULT_TABLE table=default.Table_c00510ec28ac4833831eb7a04cb5192f [id=aa4b4815e66443508e5323f8fc08377c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.591959 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cd20660f4b48658775ddf14e6cf586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.593209 758 tablet_service.cc:1468] Processing CreateTablet for tablet 99edfa13925641f3bf062b7daad5a395 (DEFAULT_TABLE table=default.Table_dec20faad45645eab2f7db58d65b3932 [id=e48fcb0bf7844fb18eb4562a6fd3dd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.593518 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99edfa13925641f3bf062b7daad5a395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.593678 886 tablet_service.cc:1468] Processing CreateTablet for tablet 24cd20660f4b48658775ddf14e6cf586 (DEFAULT_TABLE table=default.Table_c00510ec28ac4833831eb7a04cb5192f [id=aa4b4815e66443508e5323f8fc08377c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.593982 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cd20660f4b48658775ddf14e6cf586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.594188 746 tablet_service.cc:1468] Processing CreateTablet for tablet fdbeca81b90745bc82e92fad368058dd (DEFAULT_TABLE table=default.Table_9f53fde8e6ee42d5a18d464c7eac4efe [id=f7efd7d5dc5144d089e9ae273694251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.594499 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdbeca81b90745bc82e92fad368058dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.595376 889 tablet_service.cc:1468] Processing CreateTablet for tablet fdbeca81b90745bc82e92fad368058dd (DEFAULT_TABLE table=default.Table_9f53fde8e6ee42d5a18d464c7eac4efe [id=f7efd7d5dc5144d089e9ae273694251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.595705 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdbeca81b90745bc82e92fad368058dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:15.596392 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3670d1d409714821b06b98934d51762b consensus state has no leader
I20250625 13:52:15.598140 896 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8a24eb30164a46b5c3d10772476e0e (DEFAULT_TABLE table=default.Table_eaa8f84a59064ebc8833a040f29fc7bd [id=c4e0532b84b247afa4104e96a6223fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.598434 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8a24eb30164a46b5c3d10772476e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.599612 744 tablet_service.cc:1468] Processing CreateTablet for tablet ea6bd0ddae1448e78938a8ef217e15bd (DEFAULT_TABLE table=default.Table_78bde809b59a4fdc8059d75304a37e00 [id=ff2fa7eb2acf47b2a4014522a6453e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.600512 879 tablet_service.cc:1468] Processing CreateTablet for tablet ea6bd0ddae1448e78938a8ef217e15bd (DEFAULT_TABLE table=default.Table_78bde809b59a4fdc8059d75304a37e00 [id=ff2fa7eb2acf47b2a4014522a6453e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.600795 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6bd0ddae1448e78938a8ef217e15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.612803 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 965) took 5048 ms. Trace:
I20250625 13:52:15.597054 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8a24eb30164a46b5c3d10772476e0e (DEFAULT_TABLE table=default.Table_eaa8f84a59064ebc8833a040f29fc7bd [id=c4e0532b84b247afa4104e96a6223fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.614114 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8a24eb30164a46b5c3d10772476e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.617600 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6bd0ddae1448e78938a8ef217e15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.623392 2414 raft_consensus.cc:491] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.623443 1168 rpcz_store.cc:276] 0625 13:52:10.563825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.607878 (+ 44053us) service_pool.cc:225] Handling call
0625 13:52:15.560487 (+4952609us) ts_tablet_manager.cc:1145] Deleting tablet bd4cc91d274a45a393beffaaa59a35f0
0625 13:52:15.565298 (+ 4811us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.612769 (+ 47471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4952259}
I20250625 13:52:15.623546 2414 raft_consensus.cc:513] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.623899 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bd4cc91d274a45a393beffaaa59a35f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4cc91d274a45a393beffaaa59a35f0 already in progress: creating tablet
I20250625 13:52:15.624173 2414 leader_election.cc:290] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:15.619994 1171 tablet_bootstrap.cc:492] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:15.625016 985 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 6c5fc63828d9408b9cf774a966f43cf2
I20250625 13:52:15.626087 1171 tablet_bootstrap.cc:654] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.626721 1168 tablet_replica.cc:331] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.626927 1168 raft_consensus.cc:2241] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.627089 1168 raft_consensus.cc:2270] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.627641 1168 ts_tablet_manager.cc:1905] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.644449 1168 ts_tablet_manager.cc:1918] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:15.645404 2338 raft_consensus.cc:491] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.646292 2338 raft_consensus.cc:513] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.646941 2338 leader_election.cc:290] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:15.647753 1168 log.cc:1199] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/6c5fc63828d9408b9cf774a966f43cf2
W20250625 13:52:15.648241 851 leader_election.cc:343] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: dcf7fab53cae49908d2952cdaa07943a
I20250625 13:52:15.648981 1168 ts_tablet_manager.cc:1939] T 6c5fc63828d9408b9cf774a966f43cf2 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:15.649163 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b1887d3ec2f14f01808a7cecfa39064f" } new_table_name: "default.Table_3e5b4162d2ed49358c920ebbce0cfa5c" modify_external_catalogs: true
I20250625 13:52:15.650414 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 966) took 5086 ms. Trace:
I20250625 13:52:15.650689 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_390a55c23f4842618a801bfa85fce207"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.651352 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_390a55c23f4842618a801bfa85fce207 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.651343 1168 rpcz_store.cc:276] 0625 13:52:10.564097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.608010 (+ 43913us) service_pool.cc:225] Handling call
0625 13:52:15.626668 (+5018658us) ts_tablet_manager.cc:1145] Deleting tablet 6c5fc63828d9408b9cf774a966f43cf2
0625 13:52:15.626678 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.634543 (+ 7865us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.644344 (+ 9801us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.650395 (+ 6051us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5018507}
I20250625 13:52:15.651818 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 6c5fc63828d9408b9cf774a966f43cf2 (table default.Table_de6ae2243e604e0a8094e43c5b76ca27 [id=6de15b35e99646eb8eba9c11eea579c2]) successfully deleted
I20250625 13:52:15.652479 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 970) took 5017 ms. Trace:
I20250625 13:52:15.652566 1168 rpcz_store.cc:276] 0625 13:52:10.634965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.635242 (+ 277us) service_pool.cc:225] Handling call
0625 13:52:15.652347 (+5017105us) ts_tablet_manager.cc:1145] Deleting tablet 01b8bd1d9fb1470eb1bf425aa8c7af51
0625 13:52:15.652358 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.652461 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5016840}
W20250625 13:52:15.652979 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01b8bd1d9fb1470eb1bf425aa8c7af51 already in progress: creating tablet
I20250625 13:52:15.653887 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 976) took 4976 ms. Trace:
I20250625 13:52:15.654255 1168 rpcz_store.cc:276] 0625 13:52:10.677790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.702854 (+ 25064us) service_pool.cc:225] Handling call
0625 13:52:15.653762 (+4950908us) ts_tablet_manager.cc:1145] Deleting tablet c9ae350a9dbf476081dc56a709de42c8
0625 13:52:15.653774 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.653872 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4950592}
W20250625 13:52:15.655010 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c9ae350a9dbf476081dc56a709de42c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9ae350a9dbf476081dc56a709de42c8 already in progress: creating tablet
I20250625 13:52:15.655441 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 978) took 4927 ms. Trace:
I20250625 13:52:15.655835 1168 rpcz_store.cc:276] 0625 13:52:10.728456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:10.729433 (+ 977us) service_pool.cc:225] Handling call
0625 13:52:15.655344 (+4925911us) ts_tablet_manager.cc:1145] Deleting tablet 2770194711ec4a86bb1354517b095f60
0625 13:52:15.655352 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.655426 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4925674}
W20250625 13:52:15.656486 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2770194711ec4a86bb1354517b095f60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2770194711ec4a86bb1354517b095f60 already in progress: creating tablet
I20250625 13:52:15.656872 1168 tablet_replica.cc:331] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.657277 1168 raft_consensus.cc:2241] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.657696 1168 raft_consensus.cc:2270] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.658397 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bae4ae0d1a2d4285be128a582827a743" } new_table_name: "default.Table_e6384d02e4dc478baba1218c68d5765b" modify_external_catalogs: true
I20250625 13:52:15.658680 1168 ts_tablet_manager.cc:1905] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.663357 749 tablet_service.cc:1468] Processing CreateTablet for tablet 724a36167dc64f1789cbf736cede07df (DEFAULT_TABLE table=default.Table_81ed9530630f4ac9b888925f6f187d04 [id=15331889813d404d9f15fe761fd6642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.663864 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724a36167dc64f1789cbf736cede07df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.673440 896 tablet_service.cc:1468] Processing CreateTablet for tablet 724a36167dc64f1789cbf736cede07df (DEFAULT_TABLE table=default.Table_81ed9530630f4ac9b888925f6f187d04 [id=15331889813d404d9f15fe761fd6642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.673681 746 tablet_service.cc:1468] Processing CreateTablet for tablet 6cabec757d6749df9a8aae136342febf (DEFAULT_TABLE table=default.Table_603f8675b541481380889a4e72b78dbc [id=522102036bc440958f54f38777edbd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.673879 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724a36167dc64f1789cbf736cede07df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.674046 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cabec757d6749df9a8aae136342febf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.676358 750 tablet_service.cc:1468] Processing CreateTablet for tablet 25f101c4e4c1438ab6f60fa41ba938a7 (DEFAULT_TABLE table=default.Table_3d63596a7f60412eafc186aec5181d38 [id=1cea2560f77548b5b575d2cb0267c8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.676692 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f101c4e4c1438ab6f60fa41ba938a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.676702 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6cabec757d6749df9a8aae136342febf (DEFAULT_TABLE table=default.Table_603f8675b541481380889a4e72b78dbc [id=522102036bc440958f54f38777edbd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.676863 891 tablet_service.cc:1468] Processing CreateTablet for tablet 564f4e65472c4e499f26bb813029db22 (DEFAULT_TABLE table=default.Table_3e78c988506748b8b316cc58f5c6a9dc [id=cbcb375886804bae811ef489d12c667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.677008 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cabec757d6749df9a8aae136342febf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.677165 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564f4e65472c4e499f26bb813029db22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.679520 882 tablet_service.cc:1468] Processing CreateTablet for tablet 8a55df46c3d44e6cafe10e386016554f (DEFAULT_TABLE table=default.Table_51a2a2b4189c46bca8e8c16bb0b08d10 [id=cc63dadb1f464c4685dcdb53b94aeb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.679713 744 tablet_service.cc:1468] Processing CreateTablet for tablet 564f4e65472c4e499f26bb813029db22 (DEFAULT_TABLE table=default.Table_3e78c988506748b8b316cc58f5c6a9dc [id=cbcb375886804bae811ef489d12c667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.679848 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a55df46c3d44e6cafe10e386016554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.680029 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564f4e65472c4e499f26bb813029db22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.680344 748 tablet_service.cc:1468] Processing CreateTablet for tablet 1502cd79bf754827a2293a53d7c3f9b5 (DEFAULT_TABLE table=default.Table_612bc80f50694afebddc36a5f827e377 [id=4aac9020498342f2bc1062c64f01f716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.680708 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1502cd79bf754827a2293a53d7c3f9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.683665 758 tablet_service.cc:1468] Processing CreateTablet for tablet 8a55df46c3d44e6cafe10e386016554f (DEFAULT_TABLE table=default.Table_51a2a2b4189c46bca8e8c16bb0b08d10 [id=cc63dadb1f464c4685dcdb53b94aeb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.683976 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a55df46c3d44e6cafe10e386016554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.684691 889 tablet_service.cc:1468] Processing CreateTablet for tablet 25f101c4e4c1438ab6f60fa41ba938a7 (DEFAULT_TABLE table=default.Table_3d63596a7f60412eafc186aec5181d38 [id=1cea2560f77548b5b575d2cb0267c8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.685016 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f101c4e4c1438ab6f60fa41ba938a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.694770 1171 tablet_bootstrap.cc:492] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:15.696010 1170 tablet_bootstrap.cc:492] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:15.695868 1171 ts_tablet_manager.cc:1397] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.010s
I20250625 13:52:15.696506 1044 raft_consensus.cc:2466] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:15.696693 2254 ts_tablet_manager.cc:1918] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:15.696787 2254 log.cc:1199] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/df1ed922750f45eab02d1fc0c26af75e
W20250625 13:52:15.697141 983 leader_election.cc:343] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:15.697265 983 leader_election.cc:304] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:15.698379 2414 raft_consensus.cc:2747] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.698443 2411 raft_consensus.cc:491] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.698544 2411 raft_consensus.cc:513] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.699107 2411 leader_election.cc:290] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:15.697142 2254 ts_tablet_manager.cc:1939] T df1ed922750f45eab02d1fc0c26af75e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:15.700628 985 leader_election.cc:343] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 490c8902efeb46c99543747a8787c1c7
I20250625 13:52:15.700779 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3447f143704d9e835ff4f0dcff297b (DEFAULT_TABLE table=default.Table_cfe03d5988ab4502a2f04e1eef206474 [id=729d4515b71a4cd081492b5174e08635]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:15.701184 983 leader_election.cc:343] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 6c5fc63828d9408b9cf774a966f43cf2
I20250625 13:52:15.701277 983 leader_election.cc:304] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:15.701428 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 9068732806ae4ecd97e627db790b0a3d (DEFAULT_TABLE table=default.Table_a27119266aa84cf295b69911fabab0af [id=fb854116fef744de92549cab273c5195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.701553 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a0d4536a3a69448e9d9acca67393f0da (DEFAULT_TABLE table=default.Table_54c3accf07ed49259aa5cfdfcd715f72 [id=c9f77c0be7544bc7abd7269fb3fdaab8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.701676 2410 raft_consensus.cc:2747] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.701772 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 99edfa13925641f3bf062b7daad5a395 (DEFAULT_TABLE table=default.Table_dec20faad45645eab2f7db58d65b3932 [id=e48fcb0bf7844fb18eb4562a6fd3dd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.702071 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 24cd20660f4b48658775ddf14e6cf586 (DEFAULT_TABLE table=default.Table_c00510ec28ac4833831eb7a04cb5192f [id=aa4b4815e66443508e5323f8fc08377c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.702252 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "490c8902efeb46c99543747a8787c1c7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:15.702430 915 raft_consensus.cc:2466] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:15.702458 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1132) took 2699 ms. Trace:
I20250625 13:52:15.702531 2254 rpcz_store.cc:276] 0625 13:52:13.003032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.003084 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:15.501110 (+2498026us) ts_tablet_manager.cc:1145] Deleting tablet df1ed922750f45eab02d1fc0c26af75e
0625 13:52:15.501122 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.525529 (+ 24407us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.696601 (+171072us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.702443 (+ 5842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2497837}
I20250625 13:52:15.702818 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3447f143704d9e835ff4f0dcff297b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.702991 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99edfa13925641f3bf062b7daad5a395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.703042 983 leader_election.cc:304] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.703389 2411 raft_consensus.cc:2802] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:15.703456 2411 raft_consensus.cc:491] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:15.703526 2411 raft_consensus.cc:3058] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:15.704385 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cd20660f4b48658775ddf14e6cf586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.704613 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d4536a3a69448e9d9acca67393f0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.704810 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9068732806ae4ecd97e627db790b0a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.705072 2254 tablet_replica.cc:331] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:15.705243 2254 raft_consensus.cc:2241] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:15.705463 2254 pending_rounds.cc:70] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:15.705497 2254 pending_rounds.cc:77] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520653137670144 op_type: NO_OP noop_request { }
I20250625 13:52:15.705566 2254 raft_consensus.cc:2887] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:15.705634 2254 raft_consensus.cc:2270] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.705942 2411 raft_consensus.cc:513] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.706135 2254 ts_tablet_manager.cc:1905] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:15.706983 985 leader_election.cc:343] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 490c8902efeb46c99543747a8787c1c7
I20250625 13:52:15.707231 2411 leader_election.cc:290] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:15.707768 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "490c8902efeb46c99543747a8787c1c7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:15.707913 915 raft_consensus.cc:3058] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:15.710458 915 raft_consensus.cc:2466] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:15.711174 983 leader_election.cc:304] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.711902 2411 raft_consensus.cc:2802] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:15.712015 2411 raft_consensus.cc:695] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:15.712163 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1238ef9eebed4e848a72b45103c90092"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:15.712217 2411 consensus_queue.cc:237] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.712731 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1238ef9eebed4e848a72b45103c90092 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.712826 849 leader_election.cc:304] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.713201 2338 raft_consensus.cc:2802] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:15.713310 2338 raft_consensus.cc:695] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:15.713543 2338 consensus_queue.cc:237] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.715240 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 1696cbd156f94d14a5f1da76c8e58584
I20250625 13:52:15.715341 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:15.716096 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_129a669a3cfe4cffbe85f526c5d047d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.716559 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_129a669a3cfe4cffbe85f526c5d047d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.717593 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2e2711256c9042ebb651ae0ca3d02c89
I20250625 13:52:15.717748 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1696cbd156f94d14a5f1da76c8e58584 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.718264 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_53c0239319f64f789fd158d2a5fb3f12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:15.718509 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 2e2711256c9042ebb651ae0ca3d02c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:15.718740 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53c0239319f64f789fd158d2a5fb3f12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.719880 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet bc0af0cf4cb54922ad6d1f5de12a46e0
I20250625 13:52:15.720420 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_dc764e43fd094f22868df876bfcb5654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:15.720582 889 tablet_service.cc:1515] Processing DeleteTablet for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:15.720894 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc764e43fd094f22868df876bfcb5654 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.722515 2254 ts_tablet_manager.cc:1918] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:15.722648 2254 log.cc:1199] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/2404e712e18146b68757bd77362c1358
I20250625 13:52:15.724509 746 tablet_service.cc:1515] Processing DeleteTablet for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.700168 1171 raft_consensus.cc:357] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.725167 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc0af0cf4cb54922ad6d1f5de12a46e0 already in progress: creating tablet
I20250625 13:52:15.712038 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da2f8d0d9a714a28ba6fb9642d461de2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.728657 1028 tablet_service.cc:1468] Processing CreateTablet for tablet fdbeca81b90745bc82e92fad368058dd (DEFAULT_TABLE table=default.Table_9f53fde8e6ee42d5a18d464c7eac4efe [id=f7efd7d5dc5144d089e9ae273694251e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.728996 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 1696cbd156f94d14a5f1da76c8e58584 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.729414 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdbeca81b90745bc82e92fad368058dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:15.727070 849 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.730623 849 leader_election.cc:304] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.731216 2204 raft_consensus.cc:2747] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:15.731586 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 1696cbd156f94d14a5f1da76c8e58584 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1696cbd156f94d14a5f1da76c8e58584 already in progress: creating tablet
I20250625 13:52:15.732226 1171 raft_consensus.cc:383] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.733367 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 2e2711256c9042ebb651ae0ca3d02c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.733592 1171 raft_consensus.cc:738] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:52:15.733952 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2e2711256c9042ebb651ae0ca3d02c89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e2711256c9042ebb651ae0ca3d02c89 already in progress: creating tablet
I20250625 13:52:15.734244 1171 consensus_queue.cc:260] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.736232 1171 ts_tablet_manager.cc:1428] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.040s user 0.001s sys 0.002s
I20250625 13:52:15.737152 1171 tablet_bootstrap.cc:492] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:15.730896 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf7fab53cae49908d2952cdaa07943a" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:15.741223 849 leader_election.cc:343] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.741343 849 leader_election.cc:304] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:15.741719 2271 raft_consensus.cc:2747] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.743187 2254 ts_tablet_manager.cc:1939] T 2404e712e18146b68757bd77362c1358 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:15.744434 891 tablet_service.cc:1468] Processing CreateTablet for tablet 1502cd79bf754827a2293a53d7c3f9b5 (DEFAULT_TABLE table=default.Table_612bc80f50694afebddc36a5f827e377 [id=4aac9020498342f2bc1062c64f01f716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.744947 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6cabec757d6749df9a8aae136342febf (DEFAULT_TABLE table=default.Table_603f8675b541481380889a4e72b78dbc [id=522102036bc440958f54f38777edbd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.745334 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cabec757d6749df9a8aae136342febf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.745455 882 tablet_service.cc:1468] Processing CreateTablet for tablet 2fe2bf158eb64bec9e35b196703e1eb5 (DEFAULT_TABLE table=default.Table_d9bf47b9c43c456e997fd5950c43f132 [id=316e544ee11a45ea9ec3d6bc804e356b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.746528 746 tablet_service.cc:1468] Processing CreateTablet for tablet 6352e917532840ca832e32ddf0747da4 (DEFAULT_TABLE table=default.Table_ade61b1fec8c4d09a698917fe4a50bba [id=be2bd360dfd5411f96f8f9e1be4d1612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.746968 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6352e917532840ca832e32ddf0747da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.747454 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fe2bf158eb64bec9e35b196703e1eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.747465 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 564f4e65472c4e499f26bb813029db22 (DEFAULT_TABLE table=default.Table_3e78c988506748b8b316cc58f5c6a9dc [id=cbcb375886804bae811ef489d12c667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.747606 757 tablet_service.cc:1468] Processing CreateTablet for tablet c5dddb221948486080f9d0d49c907e46 (DEFAULT_TABLE table=default.Table_9f23f34708854611a2bc13f3b171ee8c [id=691370c38ef346c39926e9c464f2b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.747979 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5dddb221948486080f9d0d49c907e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.747947 758 tablet_service.cc:1468] Processing CreateTablet for tablet 20350a34b92240bfa5e26acd4a67e1bf (DEFAULT_TABLE table=default.Table_0ec681dfabc544efa1c578f0af675bb2 [id=e26550a000e44a1cb2878b883e5af8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.748263 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20350a34b92240bfa5e26acd4a67e1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.750118 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564f4e65472c4e499f26bb813029db22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.750054 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8a24eb30164a46b5c3d10772476e0e (DEFAULT_TABLE table=default.Table_eaa8f84a59064ebc8833a040f29fc7bd [id=c4e0532b84b247afa4104e96a6223fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.746181 896 tablet_service.cc:1468] Processing CreateTablet for tablet c5dddb221948486080f9d0d49c907e46 (DEFAULT_TABLE table=default.Table_9f23f34708854611a2bc13f3b171ee8c [id=691370c38ef346c39926e9c464f2b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.750530 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8a24eb30164a46b5c3d10772476e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.750705 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5dddb221948486080f9d0d49c907e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.751935 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2fe2bf158eb64bec9e35b196703e1eb5 (DEFAULT_TABLE table=default.Table_d9bf47b9c43c456e997fd5950c43f132 [id=316e544ee11a45ea9ec3d6bc804e356b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.752254 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fe2bf158eb64bec9e35b196703e1eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.754415 2398 raft_consensus.cc:491] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.754556 2398 raft_consensus.cc:513] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.754909 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1502cd79bf754827a2293a53d7c3f9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.755163 2398 leader_election.cc:290] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:15.755828 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26820eaba98401da3a8b5348be21fc6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.755908 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26820eaba98401da3a8b5348be21fc6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:15.756304 711 leader_election.cc:343] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:15.756500 711 leader_election.cc:343] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.756587 711 leader_election.cc:304] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:15.756966 2398 raft_consensus.cc:2747] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.757230 1170 tablet_bootstrap.cc:654] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.758303 1171 tablet_bootstrap.cc:654] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.759765 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1133) took 2738 ms. Trace:
I20250625 13:52:15.760082 2254 rpcz_store.cc:276] 0625 13:52:13.021272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.021318 (+ 46us) service_pool.cc:225] Handling call
0625 13:52:15.702767 (+2681449us) ts_tablet_manager.cc:1145] Deleting tablet 2404e712e18146b68757bd77362c1358
0625 13:52:15.705020 (+ 2253us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.711264 (+ 6244us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.722361 (+ 11097us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.759747 (+ 37386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2681282}
I20250625 13:52:15.760783 2254 tablet_replica.cc:331] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:15.761138 2254 raft_consensus.cc:2241] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:15.761623 2254 pending_rounds.cc:70] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:15.761883 2254 pending_rounds.cc:77] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520651730358272 op_type: NO_OP noop_request { }
I20250625 13:52:15.762092 2254 raft_consensus.cc:2887] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:15.762306 2254 raft_consensus.cc:2270] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.763244 2254 ts_tablet_manager.cc:1905] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.763636 1008 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 178) took 1049 ms. Trace:
I20250625 13:52:15.763748 1008 rpcz_store.cc:276] 0625 13:52:14.714627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.714695 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:15.763616 (+1048921us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11904}
I20250625 13:52:15.765223 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.765573 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 1502cd79bf754827a2293a53d7c3f9b5 (DEFAULT_TABLE table=default.Table_612bc80f50694afebddc36a5f827e377 [id=4aac9020498342f2bc1062c64f01f716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.765887 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1502cd79bf754827a2293a53d7c3f9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.765967 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet df1ed922750f45eab02d1fc0c26af75e (table default.Table_c7d9dab52e4c4c3bbd399e6dbc197775 [id=8bc35ca29d8d403fb6fca6bd6ce3ac66]) successfully deleted
I20250625 13:52:15.766693 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 2404e712e18146b68757bd77362c1358 (table default.Table_4dc8c488c3e44f3f882049d1c0f8feb2 [id=0532faedda28481f9742008aecb6bfd3]) successfully deleted
I20250625 13:52:15.767419 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 2e2711256c9042ebb651ae0ca3d02c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.767774 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 2fe2bf158eb64bec9e35b196703e1eb5 (DEFAULT_TABLE table=default.Table_d9bf47b9c43c456e997fd5950c43f132 [id=316e544ee11a45ea9ec3d6bc804e356b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.768118 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fe2bf158eb64bec9e35b196703e1eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.769315 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 9a2007e51d424053bc2836dedf46add2
I20250625 13:52:15.769621 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d5abd4d83514458c95ec62290b709bc3
I20250625 13:52:15.769982 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a27119266aa84cf295b69911fabab0af" } new_table_name: "default.Table_e2dd2ef2f170428bbdaa80ea2ae76e69" modify_external_catalogs: true
I20250625 13:52:15.770215 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b684a87954be44daa261986a72febecd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.770731 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b684a87954be44daa261986a72febecd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.771306 1015 tablet_service.cc:1468] Processing CreateTablet for tablet c5dddb221948486080f9d0d49c907e46 (DEFAULT_TABLE table=default.Table_9f23f34708854611a2bc13f3b171ee8c [id=691370c38ef346c39926e9c464f2b13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.771672 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5dddb221948486080f9d0d49c907e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.771799 589 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2] alter complete (version 1)
I20250625 13:52:15.771867 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 6352e917532840ca832e32ddf0747da4 (DEFAULT_TABLE table=default.Table_ade61b1fec8c4d09a698917fe4a50bba [id=be2bd360dfd5411f96f8f9e1be4d1612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.772174 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6352e917532840ca832e32ddf0747da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.772346 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a2007e51d424053bc2836dedf46add2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.772537 894 tablet_service.cc:1515] Processing DeleteTablet for tablet d5abd4d83514458c95ec62290b709bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.772658 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c75b74f21f5649bdb0ea378dd949c86d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.773110 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c75b74f21f5649bdb0ea378dd949c86d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.775336 745 tablet_service.cc:1515] Processing DeleteTablet for tablet d26820eaba98401da3a8b5348be21fc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.776340 1165 tablet_replica.cc:331] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:15.776532 1165 raft_consensus.cc:2241] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.776633 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a2007e51d424053bc2836dedf46add2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.776685 1165 raft_consensus.cc:2270] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.776927 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet d5abd4d83514458c95ec62290b709bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.777441 1165 ts_tablet_manager.cc:1905] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.779016 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.779143 2410 consensus_queue.cc:1035] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:15.779981 2271 consensus_queue.cc:1035] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:15.784511 1165 ts_tablet_manager.cc:1918] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:15.784699 1165 log.cc:1199] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d26820eaba98401da3a8b5348be21fc6
I20250625 13:52:15.785212 1165 ts_tablet_manager.cc:1939] T d26820eaba98401da3a8b5348be21fc6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:15.786127 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6352e917532840ca832e32ddf0747da4 (DEFAULT_TABLE table=default.Table_ade61b1fec8c4d09a698917fe4a50bba [id=be2bd360dfd5411f96f8f9e1be4d1612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.786618 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6352e917532840ca832e32ddf0747da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.786729 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d26820eaba98401da3a8b5348be21fc6 (table default.Table_7802866074204a24ba0302a41c6bbaf7 [id=fd00675638fa4c52b329d41cc4fa2861]) successfully deleted
I20250625 13:52:15.787062 1165 tablet_replica.cc:331] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:15.787287 1165 raft_consensus.cc:2241] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.787521 1165 raft_consensus.cc:2270] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:15.787961 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a2007e51d424053bc2836dedf46add2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.788168 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca with delete_type TABLET_DATA_DELETED (replaced by tablet 67e05cc542384b4b8117a1d9330db24b) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.788849 1165 ts_tablet_manager.cc:1905] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.789687 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 25f101c4e4c1438ab6f60fa41ba938a7 (DEFAULT_TABLE table=default.Table_3d63596a7f60412eafc186aec5181d38 [id=1cea2560f77548b5b575d2cb0267c8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.790074 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f101c4e4c1438ab6f60fa41ba938a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.792496 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 1696cbd156f94d14a5f1da76c8e58584 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.793771 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 724a36167dc64f1789cbf736cede07df (DEFAULT_TABLE table=default.Table_81ed9530630f4ac9b888925f6f187d04 [id=15331889813d404d9f15fe761fd6642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.794186 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724a36167dc64f1789cbf736cede07df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.799219 2398 raft_consensus.cc:491] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.799407 2398 raft_consensus.cc:513] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.799643 1020 tablet_service.cc:1468] Processing CreateTablet for tablet ea6bd0ddae1448e78938a8ef217e15bd (DEFAULT_TABLE table=default.Table_78bde809b59a4fdc8059d75304a37e00 [id=ff2fa7eb2acf47b2a4014522a6453e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.800035 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6bd0ddae1448e78938a8ef217e15bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.800074 2398 leader_election.cc:290] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:15.800376 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a079e985358040e78455ac8cb6e49f16 consensus state has no leader
I20250625 13:52:15.800669 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.801236 1168 ts_tablet_manager.cc:1918] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:15.801339 1168 log.cc:1199] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/87f5b36ddb474ed6903c14876b236ba8
I20250625 13:52:15.801303 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:15.801672 1168 ts_tablet_manager.cc:1939] T 87f5b36ddb474ed6903c14876b236ba8 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:15.801801 711 leader_election.cc:343] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.801947 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 988) took 4693 ms. Trace:
W20250625 13:52:15.802436 711 leader_election.cc:343] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.802560 711 leader_election.cc:304] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:15.803067 2398 raft_consensus.cc:2747] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.803587 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 8a55df46c3d44e6cafe10e386016554f (DEFAULT_TABLE table=default.Table_51a2a2b4189c46bca8e8c16bb0b08d10 [id=cc63dadb1f464c4685dcdb53b94aeb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.803988 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a55df46c3d44e6cafe10e386016554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.807194 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3775c883f8534c6d90687152b9ca22d1 (DEFAULT_TABLE table=default.Table_5027b25ff4d946e8880a10a456943c12 [id=4c9b9b3f0e1b44f7bbf415bcd3dadece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.807240 750 tablet_service.cc:1468] Processing CreateTablet for tablet 3775c883f8534c6d90687152b9ca22d1 (DEFAULT_TABLE table=default.Table_5027b25ff4d946e8880a10a456943c12 [id=4c9b9b3f0e1b44f7bbf415bcd3dadece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.807662 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3775c883f8534c6d90687152b9ca22d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.807683 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3775c883f8534c6d90687152b9ca22d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.811626 748 tablet_service.cc:1468] Processing CreateTablet for tablet e5c62f38202c44088b25dfea0270cb4b (DEFAULT_TABLE table=default.Table_83060ba51717429e8656da842eac7d78 [id=7abe1667bbd74ae5bec4191eb5ce8c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.811641 1030 tablet_service.cc:1468] Processing CreateTablet for tablet e5c62f38202c44088b25dfea0270cb4b (DEFAULT_TABLE table=default.Table_83060ba51717429e8656da842eac7d78 [id=7abe1667bbd74ae5bec4191eb5ce8c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.812139 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c62f38202c44088b25dfea0270cb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.812377 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c62f38202c44088b25dfea0270cb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.813367 758 tablet_service.cc:1515] Processing DeleteTablet for tablet d5abd4d83514458c95ec62290b709bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.815641 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 20350a34b92240bfa5e26acd4a67e1bf (DEFAULT_TABLE table=default.Table_0ec681dfabc544efa1c578f0af675bb2 [id=e26550a000e44a1cb2878b883e5af8c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.816051 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20350a34b92240bfa5e26acd4a67e1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.829243 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a6aac8f457be47e287a698c4c4df8582" } new_table_name: "default.Table_e535d4720af347929711c7e22dd487ed" modify_external_catalogs: true
I20250625 13:52:15.830276 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba1338d0562418eadabdb8dbcaedee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:15.831378 1166 tablet_bootstrap.cc:492] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.831854 1166 ts_tablet_manager.cc:1397] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.256s user 0.019s sys 0.021s
I20250625 13:52:15.836131 1166 raft_consensus.cc:357] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.841079 591 master_service.cc:948] ReplaceTablet: received request to replace tablet d3a17198b9a04138b24022cf8506ab36 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:15.863107 1166 raft_consensus.cc:383] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.863426 591 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:15.863552 1166 raft_consensus.cc:738] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.864205 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_522317d064944d7f9ab5d53e41f116f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.864756 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_522317d064944d7f9ab5d53e41f116f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.868008 1166 consensus_queue.cc:260] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:15.850687 849 consensus_peers.cc:487] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:15.828804 878 tablet_service.cc:1515] Processing DeleteTablet for tablet da2f8d0d9a714a28ba6fb9642d461de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:05 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.827805 879 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf7fab53cae49908d2952cdaa07943a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.828393 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.806089 890 tablet_service.cc:1468] Processing CreateTablet for tablet e5c62f38202c44088b25dfea0270cb4b (DEFAULT_TABLE table=default.Table_83060ba51717429e8656da842eac7d78 [id=7abe1667bbd74ae5bec4191eb5ce8c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.804550 894 tablet_service.cc:1468] Processing CreateTablet for tablet 20350a34b92240bfa5e26acd4a67e1bf (DEFAULT_TABLE table=default.Table_0ec681dfabc544efa1c578f0af675bb2 [id=e26550a000e44a1cb2878b883e5af8c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:15.871858 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8684eb9104e44227944b21784c54e496 consensus state has no leader
I20250625 13:52:15.880370 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3775c883f8534c6d90687152b9ca22d1 (DEFAULT_TABLE table=default.Table_5027b25ff4d946e8880a10a456943c12 [id=4c9b9b3f0e1b44f7bbf415bcd3dadece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.880458 758 tablet_service.cc:1468] Processing CreateTablet for tablet 3eca5d3fee7f4721b818c0c936f079c8 (DEFAULT_TABLE table=default.Table_1c8c5b5bb73e49839566884431e4b9d6 [id=a87dd3e49d08458cac8fdf2a893ca917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.881543 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eca5d3fee7f4721b818c0c936f079c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.884408 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c62f38202c44088b25dfea0270cb4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.885686 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2a80a6c4ae994eda8c3f3fb7b126616c" } new_table_name: "default.Table_26b209437c764864aae2091be187dd11" modify_external_catalogs: true
I20250625 13:52:15.887578 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20350a34b92240bfa5e26acd4a67e1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.890599 1170 tablet_bootstrap.cc:492] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:15.890751 1170 ts_tablet_manager.cc:1397] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.195s user 0.001s sys 0.004s
I20250625 13:52:15.891537 1170 raft_consensus.cc:357] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.891702 1170 raft_consensus.cc:383] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.891759 1170 raft_consensus.cc:738] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.892004 1170 consensus_queue.cc:260] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.892467 1170 ts_tablet_manager.cc:1428] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:15.892711 1170 tablet_bootstrap.cc:492] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:15.892742 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3775c883f8534c6d90687152b9ca22d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.894496 1170 tablet_bootstrap.cc:654] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.896342 1166 ts_tablet_manager.cc:1428] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20250625 13:52:15.899546 1166 tablet_bootstrap.cc:492] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:15.901293 1166 tablet_bootstrap.cc:654] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.904548 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:15.910921 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3eca5d3fee7f4721b818c0c936f079c8 (DEFAULT_TABLE table=default.Table_1c8c5b5bb73e49839566884431e4b9d6 [id=a87dd3e49d08458cac8fdf2a893ca917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.911470 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:15.913614 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eca5d3fee7f4721b818c0c936f079c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.914216 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 2de9656958164dc4ae2fb0078b38f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:15.918612 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3eca5d3fee7f4721b818c0c936f079c8 (DEFAULT_TABLE table=default.Table_1c8c5b5bb73e49839566884431e4b9d6 [id=a87dd3e49d08458cac8fdf2a893ca917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:15.919198 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eca5d3fee7f4721b818c0c936f079c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:15.864104 1171 tablet_bootstrap.cc:492] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:15.931938 1171 ts_tablet_manager.cc:1397] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.195s user 0.001s sys 0.006s
I20250625 13:52:15.932710 1171 raft_consensus.cc:357] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.932997 1171 raft_consensus.cc:383] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.933151 1171 raft_consensus.cc:738] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.933557 1171 consensus_queue.cc:260] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.934068 2398 raft_consensus.cc:491] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.934201 2398 raft_consensus.cc:513] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.934670 1171 ts_tablet_manager.cc:1428] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:15.934809 2398 leader_election.cc:290] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:15.935077 1171 tablet_bootstrap.cc:492] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:15.935791 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ca3f3760e6e41a29d004e6afe63ec98" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.936231 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ca3f3760e6e41a29d004e6afe63ec98" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:15.936888 711 leader_election.cc:343] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:15.937050 711 leader_election.cc:343] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.937121 711 leader_election.cc:304] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:15.880386 1168 rpcz_store.cc:276] 0625 13:52:11.108524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.108590 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:15.656823 (+4548233us) ts_tablet_manager.cc:1145] Deleting tablet 87f5b36ddb474ed6903c14876b236ba8
0625 13:52:15.656832 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.743558 (+ 86726us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.801140 (+ 57582us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:15.801933 (+ 793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4548025}
I20250625 13:52:15.937631 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 87f5b36ddb474ed6903c14876b236ba8 (table default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]) successfully deleted
I20250625 13:52:15.937747 2398 raft_consensus.cc:2747] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.938120 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 989) took 4794 ms. Trace:
I20250625 13:52:15.938241 1168 rpcz_store.cc:276] 0625 13:52:11.143227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.143283 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:15.938013 (+4794730us) ts_tablet_manager.cc:1145] Deleting tablet 38c954e01db443fc802d76bbbb853c14
0625 13:52:15.938022 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.938105 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4794519}
I20250625 13:52:15.938346 1166 tablet_bootstrap.cc:492] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.938460 1166 ts_tablet_manager.cc:1397] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
W20250625 13:52:15.938663 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 38c954e01db443fc802d76bbbb853c14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38c954e01db443fc802d76bbbb853c14 already in progress: creating tablet
I20250625 13:52:15.940093 1166 raft_consensus.cc:357] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.942402 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 990) took 4797 ms. Trace:
I20250625 13:52:15.942571 1168 rpcz_store.cc:276] 0625 13:52:11.144617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.144674 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:15.942281 (+4797607us) ts_tablet_manager.cc:1145] Deleting tablet 422f4d02482942dba453ff560890137b
0625 13:52:15.942292 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.942384 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4797419}
W20250625 13:52:15.943643 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 422f4d02482942dba453ff560890137b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 422f4d02482942dba453ff560890137b already in progress: creating tablet
I20250625 13:52:15.942077 2398 raft_consensus.cc:491] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:15.945834 2398 raft_consensus.cc:513] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.946404 2398 leader_election.cc:290] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:15.947044 1168 tablet_replica.cc:331] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:15.947063 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:15.947523 1168 raft_consensus.cc:2241] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:15.947620 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:15.947880 1168 raft_consensus.cc:2270] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:15.948117 711 leader_election.cc:343] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:15.948511 711 leader_election.cc:343] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:15.948611 711 leader_election.cc:304] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:15.948736 1168 ts_tablet_manager.cc:1905] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:15.948976 2398 raft_consensus.cc:2747] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:15.949201 1166 raft_consensus.cc:383] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.949374 1166 raft_consensus.cc:738] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.949762 1166 consensus_queue.cc:260] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.951553 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:15.952466 1166 ts_tablet_manager.cc:1428] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 13:52:15.957325 1166 tablet_bootstrap.cc:492] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:15.957783 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
I20250625 13:52:15.959084 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_21ce01063f674155be1463898a5a2ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:15.960497 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21ce01063f674155be1463898a5a2ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:15.962342 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:15.968555 1166 tablet_bootstrap.cc:654] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.969983 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:15.981494 1170 tablet_bootstrap.cc:492] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:15.981762 1166 tablet_bootstrap.cc:492] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:15.981773 1170 ts_tablet_manager.cc:1397] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.008s
I20250625 13:52:15.982900 1170 raft_consensus.cc:357] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.983198 1170 raft_consensus.cc:383] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.983341 1170 raft_consensus.cc:738] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.983698 1170 consensus_queue.cc:260] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.986380 1166 ts_tablet_manager.cc:1397] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250625 13:52:15.987282 1166 raft_consensus.cc:357] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.987581 1166 raft_consensus.cc:383] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:15.987694 1166 raft_consensus.cc:738] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:15.987994 1166 consensus_queue.cc:260] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:15.989308 1171 tablet_bootstrap.cc:654] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:15.990643 1166 ts_tablet_manager.cc:1428] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250625 13:52:15.999419 1170 ts_tablet_manager.cc:1428] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250625 13:52:16.001065 1166 tablet_bootstrap.cc:492] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.011157 1171 tablet_bootstrap.cc:492] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.013053 1166 tablet_bootstrap.cc:654] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.013235 1171 ts_tablet_manager.cc:1397] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.004s
I20250625 13:52:16.014775 2254 ts_tablet_manager.cc:1918] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:16.015992 1170 tablet_bootstrap.cc:492] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.016286 2254 log.cc:1199] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/b14d00c93a964edd815122e3723f89e7
I20250625 13:52:16.017159 2446 tablet.cc:1722] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:16.018424 1165 ts_tablet_manager.cc:1918] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.019254 1165 log.cc:1199] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/7506cce0ec4c4ec68f25591cb98b1252
I20250625 13:52:16.019668 1165 ts_tablet_manager.cc:1939] T 7506cce0ec4c4ec68f25591cb98b1252 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.018694 1166 tablet_bootstrap.cc:492] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.020084 1166 ts_tablet_manager.cc:1397] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20250625 13:52:16.023970 1166 raft_consensus.cc:357] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:16.029409 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9068732806ae4ecd97e627db790b0a3d consensus state has no leader
I20250625 13:52:16.030505 1166 raft_consensus.cc:383] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.030702 1166 raft_consensus.cc:738] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.030983 1166 consensus_queue.cc:260] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.031682 1166 ts_tablet_manager.cc:1428] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 13:52:16.034158 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:16.038576 2338 raft_consensus.cc:491] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.038718 2338 raft_consensus.cc:513] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.039335 2338 leader_election.cc:290] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.040395 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2de9656958164dc4ae2fb0078b38f6c3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:16.041467 849 leader_election.cc:343] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:16.041849 851 leader_election.cc:343] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2de9656958164dc4ae2fb0078b38f6c3
I20250625 13:52:16.041944 851 leader_election.cc:304] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:16.042256 2254 ts_tablet_manager.cc:1939] T b14d00c93a964edd815122e3723f89e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.042368 2338 raft_consensus.cc:2747] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.042884 2445 tablet.cc:1722] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:16.043581 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6820cb66764a482097c1ce42f9562f6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.043658 1171 raft_consensus.cc:357] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.043819 1171 raft_consensus.cc:383] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.043864 1171 raft_consensus.cc:738] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:52:16.044090 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6820cb66764a482097c1ce42f9562f6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.045333 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff57ade42e5b4619bdacc7ccacc36ae9" } modify_external_catalogs: true
I20250625 13:52:16.045468 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff57ade42e5b4619bdacc7ccacc36ae9" } modify_external_catalogs: true
I20250625 13:52:16.048413 1171 consensus_queue.cc:260] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.052920 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 74884bf313724d678d3b5f5869f09f97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:06 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.053170 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_390a55c23f4842618a801bfa85fce207" } modify_external_catalogs: true
I20250625 13:52:16.053340 1170 tablet_bootstrap.cc:654] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.054875 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_390a55c23f4842618a801bfa85fce207" } modify_external_catalogs: true
I20250625 13:52:16.060207 890 tablet_service.cc:1515] Processing DeleteTablet for tablet a30a35e3639a43c5b078169060824723 with delete_type TABLET_DATA_DELETED (replaced by tablet efc9536a716f4b9f9fc4af1637b3cd80) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:16.060529 750 tablet_service.cc:1515] Processing DeleteTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 with delete_type TABLET_DATA_DELETED (replaced by tablet 879d91dcfb0b4868a7ba6833c211e5eb) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:16.063830 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2b9e44578bd84aa6a63b1371d17d2318" } modify_external_catalogs: true
I20250625 13:52:16.064069 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2b9e44578bd84aa6a63b1371d17d2318" } modify_external_catalogs: true
I20250625 13:52:16.065872 1171 ts_tablet_manager.cc:1428] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.051s user 0.000s sys 0.002s
I20250625 13:52:16.070235 1171 tablet_bootstrap.cc:492] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.075397 1170 tablet_bootstrap.cc:492] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.075608 1170 ts_tablet_manager.cc:1397] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20250625 13:52:16.076453 1170 raft_consensus.cc:357] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.076685 1170 raft_consensus.cc:383] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.076771 1170 raft_consensus.cc:738] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.077222 1166 tablet_bootstrap.cc:492] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.078819 1166 tablet_bootstrap.cc:654] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.080713 1171 tablet_bootstrap.cc:654] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.081141 1168 ts_tablet_manager.cc:1918] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.079406 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1134) took 2918 ms. Trace:
I20250625 13:52:16.081607 2254 rpcz_store.cc:276] 0625 13:52:13.160599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.160659 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:15.760717 (+2600058us) ts_tablet_manager.cc:1145] Deleting tablet b14d00c93a964edd815122e3723f89e7
0625 13:52:15.760727 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:15.855028 (+ 94301us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.013049 (+158021us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.079387 (+ 66338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2599810}
I20250625 13:52:16.081821 1170 consensus_queue.cc:260] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.081640 1168 log.cc:1199] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/723246e1e6824faf9f9d39c3ea941fe4
I20250625 13:52:16.082158 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet b14d00c93a964edd815122e3723f89e7 (table default.Table_295dd4f121234e659090b8e4cf70faee [id=47cd45886fc94cd7ba5cafa60838d601]) successfully deleted
I20250625 13:52:16.082593 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1146) took 2654 ms. Trace:
I20250625 13:52:16.082718 2254 rpcz_store.cc:276] 0625 13:52:13.427633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.427681 (+ 48us) service_pool.cc:225] Handling call
0625 13:52:16.082457 (+2654776us) ts_tablet_manager.cc:1145] Deleting tablet 40f7656d6e53489ab982ba96bc04771b
0625 13:52:16.082467 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.082573 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654592}
W20250625 13:52:16.083222 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 40f7656d6e53489ab982ba96bc04771b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40f7656d6e53489ab982ba96bc04771b already in progress: creating tablet
I20250625 13:52:16.083669 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1145) took 2656 ms. Trace:
I20250625 13:52:16.083796 2254 rpcz_store.cc:276] 0625 13:52:13.427508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.429002 (+ 1494us) service_pool.cc:225] Handling call
0625 13:52:16.083552 (+2654550us) ts_tablet_manager.cc:1145] Deleting tablet 71170aaf26ed4f1b8c458d90db6e2765
0625 13:52:16.083560 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.083652 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654346}
W20250625 13:52:16.084200 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 71170aaf26ed4f1b8c458d90db6e2765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71170aaf26ed4f1b8c458d90db6e2765 already in progress: creating tablet
W20250625 13:52:16.084146 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 147069us
I20250625 13:52:16.084241 1168 ts_tablet_manager.cc:1939] T 723246e1e6824faf9f9d39c3ea941fe4 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.084852 1170 ts_tablet_manager.cc:1428] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250625 13:52:16.087605 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 7506cce0ec4c4ec68f25591cb98b1252 (table default.Table_78c7bbeb91584e54bef556a0af56c2a7 [id=5602ea7a0c724767b8b53d6998acef30]) successfully deleted
I20250625 13:52:16.089509 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1147) took 2650 ms. Trace:
I20250625 13:52:16.089622 2254 rpcz_store.cc:276] 0625 13:52:13.434065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.434115 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:16.084467 (+2650352us) ts_tablet_manager.cc:1145] Deleting tablet a2ebcb65c49746f3ba9aa905e657c107
0625 13:52:16.084801 (+ 334us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.084909 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2650160}
W20250625 13:52:16.090036 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a2ebcb65c49746f3ba9aa905e657c107 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ebcb65c49746f3ba9aa905e657c107 already in progress: creating tablet
I20250625 13:52:16.090467 1170 tablet_bootstrap.cc:492] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.090615 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 991) took 4941 ms. Trace:
I20250625 13:52:16.092593 1170 tablet_bootstrap.cc:654] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.091192 1168 rpcz_store.cc:276] 0625 13:52:11.148363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.148417 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:15.946988 (+4798571us) ts_tablet_manager.cc:1145] Deleting tablet 723246e1e6824faf9f9d39c3ea941fe4
0625 13:52:15.947000 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.044402 (+ 97402us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.080315 (+ 35913us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.090134 (+ 9819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4798417}
I20250625 13:52:16.094336 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 723246e1e6824faf9f9d39c3ea941fe4 (table default.Table_49756985b3fc428cbc871659e5aabdc6 [id=1178d7027158455bb95716dd9e102ef9]) successfully deleted
I20250625 13:52:16.096031 1165 tablet_replica.cc:331] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:16.096267 1165 raft_consensus.cc:2241] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.096470 1165 raft_consensus.cc:2270] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.097086 1165 ts_tablet_manager.cc:1905] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.102070 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 993) took 4944 ms. Trace:
I20250625 13:52:16.102787 1168 rpcz_store.cc:276] 0625 13:52:11.151104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.151154 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:16.095247 (+4944093us) ts_tablet_manager.cc:1145] Deleting tablet a8334d8187d2473582f0057193874b74
0625 13:52:16.095364 (+ 117us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.095853 (+ 489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4943899}
I20250625 13:52:16.090770 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1148) took 2656 ms. Trace:
I20250625 13:52:16.104468 2254 rpcz_store.cc:276] 0625 13:52:13.434175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.434365 (+ 190us) service_pool.cc:225] Handling call
0625 13:52:16.090637 (+2656272us) ts_tablet_manager.cc:1145] Deleting tablet 3713b85743d04649b27517412b3d9ea0
0625 13:52:16.090647 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.090754 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2656114}
W20250625 13:52:16.104975 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a8334d8187d2473582f0057193874b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8334d8187d2473582f0057193874b74 already in progress: creating tablet
W20250625 13:52:16.105213 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3713b85743d04649b27517412b3d9ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3713b85743d04649b27517412b3d9ea0 already in progress: creating tablet
I20250625 13:52:16.105659 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1149) took 2671 ms. Trace:
I20250625 13:52:16.105780 2254 rpcz_store.cc:276] 0625 13:52:13.434285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.434576 (+ 291us) service_pool.cc:225] Handling call
0625 13:52:16.105500 (+2670924us) ts_tablet_manager.cc:1145] Deleting tablet 569f6cba69394f2fa005686056f19929
0625 13:52:16.105540 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.105641 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2670805}
W20250625 13:52:16.106209 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 569f6cba69394f2fa005686056f19929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569f6cba69394f2fa005686056f19929 already in progress: creating tablet
I20250625 13:52:16.106590 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1153) took 2497 ms. Trace:
I20250625 13:52:16.106703 2254 rpcz_store.cc:276] 0625 13:52:13.608849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.608923 (+ 74us) service_pool.cc:225] Handling call
0625 13:52:16.106465 (+2497542us) ts_tablet_manager.cc:1145] Deleting tablet 20841d3424fb4c48b4de4ed7b2917abe
0625 13:52:16.106473 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.106574 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2486218}
W20250625 13:52:16.107074 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 20841d3424fb4c48b4de4ed7b2917abe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20841d3424fb4c48b4de4ed7b2917abe already in progress: creating tablet
I20250625 13:52:16.107429 2254 tablet_replica.cc:331] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.107676 2254 raft_consensus.cc:2241] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.107916 2254 raft_consensus.cc:2270] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.108527 2254 ts_tablet_manager.cc:1905] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.108736 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 992) took 4957 ms. Trace:
I20250625 13:52:16.109493 1166 tablet_bootstrap.cc:492] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.109640 1166 ts_tablet_manager.cc:1397] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.012s
I20250625 13:52:16.110404 1166 raft_consensus.cc:357] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.110600 1166 raft_consensus.cc:383] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.110664 1166 raft_consensus.cc:738] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.109535 1168 rpcz_store.cc:276] 0625 13:52:11.150953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.151459 (+ 506us) service_pool.cc:225] Handling call
0625 13:52:16.107889 (+4956430us) ts_tablet_manager.cc:1145] Deleting tablet 7507d50e9f464043b8901491c57391d6
0625 13:52:16.107902 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.108701 (+ 799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4955887}
I20250625 13:52:16.110878 1166 consensus_queue.cc:260] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.111393 1166 ts_tablet_manager.cc:1428] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250625 13:52:16.111529 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 7507d50e9f464043b8901491c57391d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7507d50e9f464043b8901491c57391d6 already in progress: creating tablet
I20250625 13:52:16.111694 1166 tablet_bootstrap.cc:492] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.113121 1044 raft_consensus.cc:3058] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:16.113348 1166 tablet_bootstrap.cc:654] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.129952 1168 tablet_replica.cc:331] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:16.131450 1170 tablet_bootstrap.cc:492] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.132054 1170 ts_tablet_manager.cc:1397] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20250625 13:52:16.132844 1168 raft_consensus.cc:2241] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:16.133059 1170 raft_consensus.cc:357] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.134168 1170 raft_consensus.cc:383] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.134371 1170 raft_consensus.cc:738] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.133860 890 tablet_service.cc:1515] Processing DeleteTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:16.134904 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9545d17bca8440d5894a77ccd3fafc1b" } modify_external_catalogs: true
I20250625 13:52:16.135098 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9545d17bca8440d5894a77ccd3fafc1b" } modify_external_catalogs: true
I20250625 13:52:16.138471 1170 consensus_queue.cc:260] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:16.141064 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0a13c34b24244d58daa0d67589b5100 consensus state has no leader
I20250625 13:52:16.141429 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_753dc763991a4d2c900b8f1dc5e6138a" } modify_external_catalogs: true
I20250625 13:52:16.141697 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_753dc763991a4d2c900b8f1dc5e6138a" } modify_external_catalogs: true
I20250625 13:52:16.143297 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3642d962b3544fc58d26f3d5592639b9" } new_table_name: "default.Table_08c66d2c51c4482f905a1026a08ab47f" modify_external_catalogs: true
I20250625 13:52:16.145831 2398 raft_consensus.cc:491] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.145972 2398 raft_consensus.cc:513] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.146078 1170 ts_tablet_manager.cc:1428] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250625 13:52:16.146600 2398 leader_election.cc:290] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.147082 1170 tablet_bootstrap.cc:492] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.147053 1168 pending_rounds.cc:70] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:16.147185 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "413afd7b62d24fe8b9eceadd57e29fc8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:16.148138 711 leader_election.cc:343] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.149053 1170 tablet_bootstrap.cc:654] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.150077 1165 ts_tablet_manager.cc:1918] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.147631 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "413afd7b62d24fe8b9eceadd57e29fc8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.150369 1168 pending_rounds.cc:77] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520658284638208 op_type: NO_OP noop_request { }
I20250625 13:52:16.150496 1168 raft_consensus.cc:2887] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:16.150616 1168 raft_consensus.cc:2270] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.148336 1171 tablet_bootstrap.cc:492] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.141448 2254 ts_tablet_manager.cc:1918] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:16.151746 2254 log.cc:1199] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/6d95a376304e464ab75e2e1e00e7a924
I20250625 13:52:16.152287 2254 ts_tablet_manager.cc:1939] T 6d95a376304e464ab75e2e1e00e7a924 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.151742 1171 ts_tablet_manager.cc:1397] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.006s
I20250625 13:52:16.152853 2271 raft_consensus.cc:491] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.152973 2271 raft_consensus.cc:513] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.153033 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1159) took 2512 ms. Trace:
I20250625 13:52:16.153110 2254 rpcz_store.cc:276] 0625 13:52:13.640611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.640665 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:16.107378 (+2466713us) ts_tablet_manager.cc:1145] Deleting tablet 6d95a376304e464ab75e2e1e00e7a924
0625 13:52:16.107389 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.118730 (+ 11341us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.141338 (+ 22608us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.153016 (+ 11678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2416680}
W20250625 13:52:16.150795 711 leader_election.cc:343] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.153359 711 leader_election.cc:304] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.153493 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1158) took 2519 ms. Trace:
I20250625 13:52:16.153530 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]) successfully deleted
I20250625 13:52:16.153553 2254 rpcz_store.cc:276] 0625 13:52:13.633602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.633640 (+ 38us) service_pool.cc:225] Handling call
0625 13:52:16.153372 (+2519732us) ts_tablet_manager.cc:1145] Deleting tablet 9f24ae7c08bd42959ab48f7007ec98d3
0625 13:52:16.153383 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.153479 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2442885}
I20250625 13:52:16.153895 2398 raft_consensus.cc:2747] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:16.154073 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9f24ae7c08bd42959ab48f7007ec98d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f24ae7c08bd42959ab48f7007ec98d3 already in progress: creating tablet
I20250625 13:52:16.154839 2271 leader_election.cc:290] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.155422 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a30a35e3639a43c5b078169060824723" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.155226 1165 log.cc:1199] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/9a2007e51d424053bc2836dedf46add2
W20250625 13:52:16.155910 849 leader_election.cc:343] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:16.156838 851 leader_election.cc:343] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a30a35e3639a43c5b078169060824723
I20250625 13:52:16.156945 851 leader_election.cc:304] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:16.157393 2271 raft_consensus.cc:2747] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.157344 1171 raft_consensus.cc:357] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.157485 1171 raft_consensus.cc:383] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.157529 1171 raft_consensus.cc:738] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.157703 1171 consensus_queue.cc:260] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.158178 1171 ts_tablet_manager.cc:1428] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20250625 13:52:16.159508 1168 ts_tablet_manager.cc:1905] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.159548 2254 tablet_replica.cc:331] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.159710 2254 raft_consensus.cc:2241] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.160754 2254 raft_consensus.cc:2270] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.160974 1165 ts_tablet_manager.cc:1939] T 9a2007e51d424053bc2836dedf46add2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.161782 2254 ts_tablet_manager.cc:1905] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.169265 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 9a2007e51d424053bc2836dedf46add2 (table default.Table_4a226ca5c5c84ef380d6c37dd08f6740 [id=fa06a5b0571b4a3a8b9b68f1f05fc3e6]) successfully deleted
I20250625 13:52:16.171273 1165 tablet_replica.cc:331] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:52:16.178943 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a33c304d652e4e61a3bcfb3ab7067244 consensus state has no leader
I20250625 13:52:16.180275 1170 tablet_bootstrap.cc:492] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.180505 1170 ts_tablet_manager.cc:1397] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.003s
I20250625 13:52:16.181372 1170 raft_consensus.cc:357] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.181648 1170 raft_consensus.cc:383] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.181650 1171 tablet_bootstrap.cc:492] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.181768 1170 raft_consensus.cc:738] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.182051 1170 consensus_queue.cc:260] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.182722 1170 ts_tablet_manager.cc:1428] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:16.183017 1171 tablet_bootstrap.cc:654] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.174325 1165 raft_consensus.cc:2241] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.186359 1165 raft_consensus.cc:2270] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.188272 1170 tablet_bootstrap.cc:492] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.188933 2254 ts_tablet_manager.cc:1918] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.190225 2254 log.cc:1199] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ac4d368347444846880954ced9651404
I20250625 13:52:16.190268 1165 ts_tablet_manager.cc:1905] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.190984 2254 ts_tablet_manager.cc:1939] T ac4d368347444846880954ced9651404 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.191465 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1162) took 2435 ms. Trace:
I20250625 13:52:16.192008 2254 rpcz_store.cc:276] 0625 13:52:13.755866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.759143 (+ 3277us) service_pool.cc:225] Handling call
0625 13:52:16.159488 (+2400345us) ts_tablet_manager.cc:1145] Deleting tablet ac4d368347444846880954ced9651404
0625 13:52:16.159509 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.171412 (+ 11903us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.188831 (+ 17419us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.191448 (+ 2617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2400119}
I20250625 13:52:16.192727 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ac4d368347444846880954ced9651404 (table default.Table_581fcddbe9bb428d84556e6156637b4a [id=3a7ba6e0a96c4f3fb656f9ac4f6ee82d]) successfully deleted
I20250625 13:52:16.193166 2254 tablet_replica.cc:331] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.193598 2254 raft_consensus.cc:2241] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:16.194216 2254 raft_consensus.cc:2270] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.199769 2254 ts_tablet_manager.cc:1905] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.202088 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0da0d45b112845ca956fc3f260a6901d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.202631 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0da0d45b112845ca956fc3f260a6901d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.207391 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:16.211112 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d74b5058e9b149069173bc5409364d6c" } new_table_name: "default.Table_050b3df547a443f797e988c8fcb3dc97" modify_external_catalogs: true
I20250625 13:52:16.215576 2456 tablet.cc:1722] T 20c12f55ed5b4425aea1bf45cdc0658e P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:16.216184 1166 tablet_bootstrap.cc:492] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.216403 1166 ts_tablet_manager.cc:1397] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.014s
I20250625 13:52:16.217326 1166 raft_consensus.cc:357] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.217592 1166 raft_consensus.cc:383] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.217695 1166 raft_consensus.cc:738] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.218014 1166 consensus_queue.cc:260] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.218608 1166 ts_tablet_manager.cc:1428] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:16.218946 1166 tablet_bootstrap.cc:492] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.221516 1166 tablet_bootstrap.cc:654] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.230125 1168 ts_tablet_manager.cc:1918] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:16.231281 1168 log.cc:1199] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/7ba1338d0562418eadabdb8dbcaedee2
I20250625 13:52:16.231981 2254 ts_tablet_manager.cc:1918] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:16.232146 2254 log.cc:1199] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/38a23b08a03b48eaa051f670eb7dac65
I20250625 13:52:16.232576 2254 ts_tablet_manager.cc:1939] T 38a23b08a03b48eaa051f670eb7dac65 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.232959 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1163) took 2448 ms. Trace:
I20250625 13:52:16.233934 1166 tablet_bootstrap.cc:492] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.234154 1166 ts_tablet_manager.cc:1397] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250625 13:52:16.234437 1168 ts_tablet_manager.cc:1939] T 7ba1338d0562418eadabdb8dbcaedee2 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.234983 1166 raft_consensus.cc:357] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.235226 1166 raft_consensus.cc:383] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.235342 1166 raft_consensus.cc:738] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.235626 1166 consensus_queue.cc:260] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.236183 1166 ts_tablet_manager.cc:1428] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:16.238632 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5d467dbf133147e7a517c6fc4dff433a
I20250625 13:52:16.240178 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d467dbf133147e7a517c6fc4dff433a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:16.240646 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d467dbf133147e7a517c6fc4dff433a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.240873 1166 tablet_bootstrap.cc:492] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.233155 2254 rpcz_store.cc:276] 0625 13:52:13.784742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.786891 (+ 2149us) service_pool.cc:225] Handling call
0625 13:52:16.193117 (+2406226us) ts_tablet_manager.cc:1145] Deleting tablet 38a23b08a03b48eaa051f670eb7dac65
0625 13:52:16.193128 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.206104 (+ 12976us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.231884 (+ 25780us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.232941 (+ 1057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406040}
I20250625 13:52:16.242822 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d467dbf133147e7a517c6fc4dff433a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:16.242818 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 38a23b08a03b48eaa051f670eb7dac65 (table default.Table_91567a207fcd4540937c0e047ea86de5 [id=5fcd0958c61942289b4eb28c077ceec3]) successfully deleted
I20250625 13:52:16.243209 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1165) took 2418 ms. Trace:
I20250625 13:52:16.243357 2254 rpcz_store.cc:276] 0625 13:52:13.824966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.825024 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:16.243075 (+2418051us) ts_tablet_manager.cc:1145] Deleting tablet 55a254d82f764aa2860e465b4857a413
0625 13:52:16.243088 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.243192 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417602}
W20250625 13:52:16.243893 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 55a254d82f764aa2860e465b4857a413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55a254d82f764aa2860e465b4857a413 already in progress: creating tablet
I20250625 13:52:16.244330 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1166) took 2417 ms. Trace:
I20250625 13:52:16.244447 2254 rpcz_store.cc:276] 0625 13:52:13.826792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.826844 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:16.244212 (+2417368us) ts_tablet_manager.cc:1145] Deleting tablet 6c331c57bff44091be2849c21d6ab2d7
0625 13:52:16.244221 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.244313 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417177}
W20250625 13:52:16.244922 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6c331c57bff44091be2849c21d6ab2d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c331c57bff44091be2849c21d6ab2d7 already in progress: creating tablet
I20250625 13:52:16.245296 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1164) took 2423 ms. Trace:
I20250625 13:52:16.245656 1166 tablet_bootstrap.cc:654] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.245752 2254 rpcz_store.cc:276] 0625 13:52:13.821370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.827678 (+ 6308us) service_pool.cc:225] Handling call
0625 13:52:16.245200 (+2417522us) ts_tablet_manager.cc:1145] Deleting tablet d35517bbf1cb4695ae46094dc883175f
0625 13:52:16.245209 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.245282 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417354}
I20250625 13:52:16.245553 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1004) took 4903 ms. Trace:
W20250625 13:52:16.250151 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d35517bbf1cb4695ae46094dc883175f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35517bbf1cb4695ae46094dc883175f already in progress: creating tablet
I20250625 13:52:16.251093 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1167) took 2407 ms. Trace:
I20250625 13:52:16.251245 2254 rpcz_store.cc:276] 0625 13:52:13.843659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.843712 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:16.250971 (+2407259us) ts_tablet_manager.cc:1145] Deleting tablet 6fcc17f839504d6b85526b638763ec5b
0625 13:52:16.250981 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.251073 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407062}
W20250625 13:52:16.251796 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6fcc17f839504d6b85526b638763ec5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fcc17f839504d6b85526b638763ec5b already in progress: creating tablet
I20250625 13:52:16.246340 1168 rpcz_store.cc:276] 0625 13:52:11.341352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.341408 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:16.129853 (+4788445us) ts_tablet_manager.cc:1145] Deleting tablet 7ba1338d0562418eadabdb8dbcaedee2
0625 13:52:16.129885 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.177053 (+ 47168us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.229558 (+ 52505us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.245237 (+ 15679us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4751392}
I20250625 13:52:16.253507 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 7ba1338d0562418eadabdb8dbcaedee2 (table default.Table_c7891a214410484c907ee024a3f40331 [id=21a2a642fff348518e79d4e4db1f32ad]) successfully deleted
I20250625 13:52:16.259616 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1010) took 4839 ms. Trace:
I20250625 13:52:16.259811 1168 rpcz_store.cc:276] 0625 13:52:11.419873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.419917 (+ 44us) service_pool.cc:225] Handling call
0625 13:52:16.259492 (+4839575us) ts_tablet_manager.cc:1145] Deleting tablet 5d2ce669ecd441939be8e8c71eea7bfa
0625 13:52:16.259502 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.259595 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4839405}
W20250625 13:52:16.260492 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 5d2ce669ecd441939be8e8c71eea7bfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2ce669ecd441939be8e8c71eea7bfa already in progress: creating tablet
I20250625 13:52:16.262130 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1011) took 4841 ms. Trace:
I20250625 13:52:16.263051 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1168) took 2404 ms. Trace:
I20250625 13:52:16.263197 2254 rpcz_store.cc:276] 0625 13:52:13.858219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.858281 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:16.252117 (+2393836us) ts_tablet_manager.cc:1145] Deleting tablet a7b72345e04445f7a393c469a348d126
0625 13:52:16.252126 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.263028 (+ 10902us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393654}
I20250625 13:52:16.262706 1165 ts_tablet_manager.cc:1918] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.263095 1168 rpcz_store.cc:276] 0625 13:52:11.420385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.420430 (+ 45us) service_pool.cc:225] Handling call
0625 13:52:16.261736 (+4841306us) ts_tablet_manager.cc:1145] Deleting tablet 12a7db105cb34bb2ac7a411256c43cae
0625 13:52:16.261755 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.261952 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4841029}
I20250625 13:52:16.264528 1165 log.cc:1199] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/8e2bc4df39674dc3be0cd32c68f1ebca
W20250625 13:52:16.265518 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 12a7db105cb34bb2ac7a411256c43cae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12a7db105cb34bb2ac7a411256c43cae already in progress: creating tablet
I20250625 13:52:16.266444 1168 tablet_replica.cc:331] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:16.267612 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a7b72345e04445f7a393c469a348d126 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7b72345e04445f7a393c469a348d126 already in progress: creating tablet
I20250625 13:52:16.267670 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 86af0ea6648045a1ac4ba903d20849b7 (table default.Table_fac1b7cf47334e72b5815c78f0e35ba6 [id=65ec4be481574b20a64c98a75c6c4367]). Expected version 4 got 0
I20250625 13:52:16.267880 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 20c12f55ed5b4425aea1bf45cdc0658e (table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]). Expected version 1 got 0
I20250625 13:52:16.267916 2254 tablet_replica.cc:331] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.268101 2254 raft_consensus.cc:2241] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:16.268431 2254 raft_consensus.cc:2270] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.268370 1165 ts_tablet_manager.cc:1939] T 8e2bc4df39674dc3be0cd32c68f1ebca P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.268573 1168 raft_consensus.cc:2241] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.270998 1168 raft_consensus.cc:2270] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.272590 2254 ts_tablet_manager.cc:1905] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.272759 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_19cabef12f194c91ab805ee4d346fbae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.273267 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19cabef12f194c91ab805ee4d346fbae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.275502 1168 ts_tablet_manager.cc:1905] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.278110 593 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ccca6a2c228d40b999fbf35b20c5c575
I20250625 13:52:16.280710 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 8e2bc4df39674dc3be0cd32c68f1ebca (table default.Table_ac4d598c675a46d1bbd2a08d9652f2c7 [id=b36fcb27ef404ffeaa1532ad1063dbfe]) successfully deleted
I20250625 13:52:16.280786 1165 tablet_replica.cc:331] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
W20250625 13:52:16.281172 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d5abd4d83514458c95ec62290b709bc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5abd4d83514458c95ec62290b709bc3 already in progress: creating tablet
I20250625 13:52:16.283336 1165 raft_consensus.cc:2241] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.283906 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 9514bbbda0ba4329a18b0a42e635c6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.283906 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 37e4bd4aaab143ffa65a668a952ae345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.284175 1165 raft_consensus.cc:2270] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.284529 593 master_service.cc:948] ReplaceTablet: received request to replace tablet e38a85c299f04d60a3c465571fd33971 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:16.285171 1165 ts_tablet_manager.cc:1905] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.286696 1166 tablet_bootstrap.cc:492] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.291445 1166 ts_tablet_manager.cc:1397] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.002s
I20250625 13:52:16.297128 2254 ts_tablet_manager.cc:1918] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:16.303395 1166 raft_consensus.cc:357] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.304061 1166 raft_consensus.cc:383] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.304214 1166 raft_consensus.cc:738] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.304483 1166 consensus_queue.cc:260] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.305089 1166 ts_tablet_manager.cc:1428] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250625 13:52:16.307000 2254 log.cc:1199] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/87f5b36ddb474ed6903c14876b236ba8
I20250625 13:52:16.309597 2254 ts_tablet_manager.cc:1939] T 87f5b36ddb474ed6903c14876b236ba8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.310007 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1169) took 2317 ms. Trace:
I20250625 13:52:16.310154 2254 rpcz_store.cc:276] 0625 13:52:13.992394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.992446 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:16.267864 (+2275418us) ts_tablet_manager.cc:1145] Deleting tablet 87f5b36ddb474ed6903c14876b236ba8
0625 13:52:16.267875 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.294047 (+ 26172us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.297041 (+ 2994us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.309990 (+ 12949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2275182}
I20250625 13:52:16.310537 1168 ts_tablet_manager.cc:1918] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.310720 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 87f5b36ddb474ed6903c14876b236ba8 (table default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]) successfully deleted
I20250625 13:52:16.311092 2254 tablet_replica.cc:331] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.310995 1168 log.cc:1199] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/74884bf313724d678d3b5f5869f09f97
I20250625 13:52:16.311369 2204 raft_consensus.cc:491] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.311491 2204 raft_consensus.cc:513] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.312096 2204 leader_election.cc:290] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.313848 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69672579a83c41d09ee7c78a6fa7d746" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.314401 1166 tablet_bootstrap.cc:492] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:16.314463 851 leader_election.cc:343] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 69672579a83c41d09ee7c78a6fa7d746
W20250625 13:52:16.314867 849 leader_election.cc:343] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.314961 849 leader_election.cc:304] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:16.315553 2338 raft_consensus.cc:2747] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.316126 1166 tablet_bootstrap.cc:654] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:16.317067 849 consensus_peers.cc:487] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:16.318814 1165 ts_tablet_manager.cc:1918] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.318909 1165 log.cc:1199] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/1ca3f3760e6e41a29d004e6afe63ec98
I20250625 13:52:16.319201 1165 ts_tablet_manager.cc:1939] T 1ca3f3760e6e41a29d004e6afe63ec98 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.319784 2338 raft_consensus.cc:491] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.319903 2338 raft_consensus.cc:513] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.320211 2254 raft_consensus.cc:2241] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.320371 2254 raft_consensus.cc:2270] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.320497 2338 leader_election.cc:290] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.320875 1166 tablet_bootstrap.cc:492] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.320953 2254 ts_tablet_manager.cc:1905] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.321115 1166 ts_tablet_manager.cc:1397] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
W20250625 13:52:16.321516 851 leader_election.cc:343] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: dfaba56442c0498da95e6fad126ab86d
I20250625 13:52:16.322294 1166 raft_consensus.cc:357] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.323652 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 1ca3f3760e6e41a29d004e6afe63ec98 (table default.Table_e7bc36f60d844f0d9d27755d6f58f63b [id=0a7bf2d0fdb148d2a201bddf8702be9b]) successfully deleted
I20250625 13:52:16.325147 1166 raft_consensus.cc:383] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.325281 1166 raft_consensus.cc:738] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.325564 1166 consensus_queue.cc:260] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.326153 1166 ts_tablet_manager.cc:1428] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:16.326669 1166 tablet_bootstrap.cc:492] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.326397 1165 tablet_replica.cc:331] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:16.327406 1168 ts_tablet_manager.cc:1939] T 74884bf313724d678d3b5f5869f09f97 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.327510 1165 raft_consensus.cc:2241] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.327648 1165 raft_consensus.cc:2270] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.330585 1165 ts_tablet_manager.cc:1905] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.321751 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfaba56442c0498da95e6fad126ab86d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:16.332648 849 leader_election.cc:343] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.332773 849 leader_election.cc:304] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:16.333267 2204 raft_consensus.cc:2747] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.309160 1170 tablet_bootstrap.cc:654] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.334697 1166 tablet_bootstrap.cc:654] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.330156 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1012) took 4909 ms. Trace:
I20250625 13:52:16.335394 2204 raft_consensus.cc:491] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.335845 2204 raft_consensus.cc:513] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.336057 1171 tablet_bootstrap.cc:492] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.336177 1171 ts_tablet_manager.cc:1397] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.155s user 0.018s sys 0.023s
I20250625 13:52:16.336472 2204 leader_election.cc:290] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:16.336839 1171 raft_consensus.cc:357] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.337013 1171 raft_consensus.cc:383] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.337080 1171 raft_consensus.cc:738] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.337327 1171 consensus_queue.cc:260] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:16.337430 851 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:52:16.337970 1171 ts_tablet_manager.cc:1428] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:16.338075 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c605d5b64c164c1594b0a40cfa2c4cf6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:16.338559 849 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.338645 849 leader_election.cc:304] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:16.339031 2338 raft_consensus.cc:2747] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.343869 1168 rpcz_store.cc:276] 0625 13:52:11.420853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.420898 (+ 45us) service_pool.cc:225] Handling call
0625 13:52:16.266180 (+4845282us) ts_tablet_manager.cc:1145] Deleting tablet 74884bf313724d678d3b5f5869f09f97
0625 13:52:16.266189 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.288533 (+ 22344us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.310133 (+ 21600us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.330045 (+ 19912us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4845105}
I20250625 13:52:16.345244 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b] alter complete (version 3)
I20250625 13:52:16.346138 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ab7202a8761444a0a731993d890fcdeb
I20250625 13:52:16.346123 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:16.346494 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 74884bf313724d678d3b5f5869f09f97 (table default.Table_25d9c73fdf1b43b787345769f54915d6 [id=79ace8a081c447a5b4c44b5911a906c8]) successfully deleted
I20250625 13:52:16.346693 750 tablet_service.cc:1515] Processing DeleteTablet for tablet ab7202a8761444a0a731993d890fcdeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:16.347065 890 tablet_service.cc:1515] Processing DeleteTablet for tablet ab7202a8761444a0a731993d890fcdeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:16.347066 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ab7202a8761444a0a731993d890fcdeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.347772 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a57268a18f4f00a39f8360f9c90737 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:16.351114 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1009) took 4931 ms. Trace:
I20250625 13:52:16.351523 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 23fa1b7e7c13444fb8ee07807dabd813
I20250625 13:52:16.352067 1171 tablet_bootstrap.cc:492] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.351557 1168 rpcz_store.cc:276] 0625 13:52:11.419581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.521535 (+101954us) service_pool.cc:225] Handling call
0625 13:52:16.350518 (+4828983us) ts_tablet_manager.cc:1145] Deleting tablet af704cdf44ef42f7b5ea10b199d0f017
0625 13:52:16.350694 (+ 176us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.351096 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4828697}
W20250625 13:52:16.353374 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet af704cdf44ef42f7b5ea10b199d0f017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af704cdf44ef42f7b5ea10b199d0f017 already in progress: creating tablet
I20250625 13:52:16.353539 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 23fa1b7e7c13444fb8ee07807dabd813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.353732 1171 tablet_bootstrap.cc:654] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.353776 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 23fa1b7e7c13444fb8ee07807dabd813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:16.354166 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1018) took 4811 ms. Trace:
I20250625 13:52:16.354818 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 23fa1b7e7c13444fb8ee07807dabd813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:16.354681 1168 rpcz_store.cc:276] 0625 13:52:11.542653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.542707 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:16.353370 (+4810663us) ts_tablet_manager.cc:1145] Deleting tablet 4acfcd35d6434ee59b9b0d831a59938c
0625 13:52:16.353378 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.354148 (+ 770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4810456}
W20250625 13:52:16.356182 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4acfcd35d6434ee59b9b0d831a59938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4acfcd35d6434ee59b9b0d831a59938c already in progress: creating tablet
I20250625 13:52:16.357584 1166 tablet_bootstrap.cc:492] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.357704 1166 ts_tablet_manager.cc:1397] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20250625 13:52:16.358371 1166 raft_consensus.cc:357] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.358528 1166 raft_consensus.cc:383] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.358579 1166 raft_consensus.cc:738] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.358775 1166 consensus_queue.cc:260] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.359246 1166 ts_tablet_manager.cc:1428] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 13:52:16.359736 1166 tablet_bootstrap.cc:492] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.362566 2398 raft_consensus.cc:491] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.362690 2398 raft_consensus.cc:513] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.363278 2398 leader_election.cc:290] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:16.363929 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db6e9db2375144b38ec43ffca9ca4c1e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:16.364440 711 leader_election.cc:343] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.364928 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db6e9db2375144b38ec43ffca9ca4c1e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:16.365381 711 leader_election.cc:343] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.365473 711 leader_election.cc:304] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.365861 2398 raft_consensus.cc:2747] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.368327 1168 tablet_replica.cc:331] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:16.369119 1168 raft_consensus.cc:2241] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.369855 1168 raft_consensus.cc:2270] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.370160 1166 tablet_bootstrap.cc:654] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.369930 1165 ts_tablet_manager.cc:1918] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.370910 1165 log.cc:1199] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d8985f259eb54ffdab29a9f6a2c5ae16
I20250625 13:52:16.372570 1171 tablet_bootstrap.cc:492] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.372924 1171 ts_tablet_manager.cc:1397] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20250625 13:52:16.374220 1171 raft_consensus.cc:357] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.374439 1171 raft_consensus.cc:383] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.374554 1171 raft_consensus.cc:738] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.374708 1165 ts_tablet_manager.cc:1939] T d8985f259eb54ffdab29a9f6a2c5ae16 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.374919 1171 consensus_queue.cc:260] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.375510 1171 ts_tablet_manager.cc:1428] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:16.377637 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d8985f259eb54ffdab29a9f6a2c5ae16 (table default.Table_d46618ecfae24fe2bdd9fc2feea12ad5 [id=5b767eccaaaa46e0a054dd083e181cbd]) successfully deleted
W20250625 13:52:16.378815 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 5d467dbf133147e7a517c6fc4dff433a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d467dbf133147e7a517c6fc4dff433a already in progress: creating tablet
I20250625 13:52:16.379611 1165 tablet_replica.cc:331] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:16.372574 1168 ts_tablet_manager.cc:1905] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.386931 1171 tablet_bootstrap.cc:492] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.390444 1010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 180) took 1092 ms. Trace:
I20250625 13:52:16.391532 1165 raft_consensus.cc:2241] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.391718 1165 raft_consensus.cc:2270] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.392377 1170 tablet_bootstrap.cc:492] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.392525 1170 ts_tablet_manager.cc:1397] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.205s user 0.023s sys 0.002s
I20250625 13:52:16.393263 1170 raft_consensus.cc:357] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.393447 1170 raft_consensus.cc:383] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.393498 1170 raft_consensus.cc:738] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.393738 1170 consensus_queue.cc:260] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.393967 1165 ts_tablet_manager.cc:1905] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.394775 1170 ts_tablet_manager.cc:1428] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:16.395483 1170 tablet_bootstrap.cc:492] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.400395 1165 ts_tablet_manager.cc:1918] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.402854 1171 tablet_bootstrap.cc:654] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.404114 1170 tablet_bootstrap.cc:654] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.404739 1168 ts_tablet_manager.cc:1918] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:16.404919 1165 log.cc:1199] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/413afd7b62d24fe8b9eceadd57e29fc8
I20250625 13:52:16.405329 1165 ts_tablet_manager.cc:1939] T 413afd7b62d24fe8b9eceadd57e29fc8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.408382 1165 tablet_replica.cc:331] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:16.409148 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 413afd7b62d24fe8b9eceadd57e29fc8 (table default.Table_12771e585c3348049cc0ab4bf2a95935 [id=5eed1277a07641b6abc65ef047b31615]) successfully deleted
I20250625 13:52:16.410115 1165 raft_consensus.cc:2241] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.412492 1165 raft_consensus.cc:2270] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.413125 1165 ts_tablet_manager.cc:1905] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.414299 2254 ts_tablet_manager.cc:1918] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:16.417863 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54cc2d0cf95b41d6b36483af5436fcde consensus state has no leader
I20250625 13:52:16.418752 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 882152a3a5a2458d803ba7f75b31bc30
I20250625 13:52:16.419216 1165 ts_tablet_manager.cc:1918] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:16.421602 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ec4be031b7ee44a59b7015fda0f6bf44" } new_table_name: "default.Table_db68f5b8cfc64881a64265aa1240324e" modify_external_catalogs: true
I20250625 13:52:16.425470 589 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
I20250625 13:52:16.426297 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7344157e71594339a43c2dd8d6e1fe58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.426781 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7344157e71594339a43c2dd8d6e1fe58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.427618 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d3f4f37b0b30453789b651b77c3b3833"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.428014 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3f4f37b0b30453789b651b77c3b3833 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.428313 2254 log.cc:1199] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d43ba17ebfec4244894d1b6e23200c33
I20250625 13:52:16.428684 2254 ts_tablet_manager.cc:1939] T d43ba17ebfec4244894d1b6e23200c33 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.429003 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1170) took 2372 ms. Trace:
I20250625 13:52:16.429946 2254 rpcz_store.cc:276] 0625 13:52:14.056499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.056548 (+ 49us) service_pool.cc:225] Handling call
0625 13:52:16.311036 (+2254488us) ts_tablet_manager.cc:1145] Deleting tablet d43ba17ebfec4244894d1b6e23200c33
0625 13:52:16.311049 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.406384 (+ 95335us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.414185 (+ 7801us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.428985 (+ 14800us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254319}
I20250625 13:52:16.430738 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet d43ba17ebfec4244894d1b6e23200c33 (table default.Table_bc6c1303fba14b97b873dd23a423bf8c [id=999e931fc6724eec9b6d8bb94bd3cf23]) successfully deleted
I20250625 13:52:16.431259 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1171) took 2300 ms. Trace:
I20250625 13:52:16.431581 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_74bb2a80612b44d5b356dcd8ce9a1fde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.409814 2204 consensus_queue.cc:1035] T b54183cf36b049c7bed30deb1604c59a P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:52:16.431985 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74bb2a80612b44d5b356dcd8ce9a1fde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.431923 2254 rpcz_store.cc:276] 0625 13:52:14.130547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.130601 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:16.431105 (+2300504us) ts_tablet_manager.cc:1145] Deleting tablet 19f1050a29b540558aaf1e31118e4531
0625 13:52:16.431117 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.431223 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2300329}
W20250625 13:52:16.432320 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 19f1050a29b540558aaf1e31118e4531 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f1050a29b540558aaf1e31118e4531 already in progress: creating tablet
I20250625 13:52:16.419848 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 882152a3a5a2458d803ba7f75b31bc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:16.432758 1168 log.cc:1199] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/38a23b08a03b48eaa051f670eb7dac65
I20250625 13:52:16.433218 1168 ts_tablet_manager.cc:1939] T 38a23b08a03b48eaa051f670eb7dac65 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.434134 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1019) took 4882 ms. Trace:
I20250625 13:52:16.434270 1168 rpcz_store.cc:276] 0625 13:52:11.551390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.552858 (+ 1468us) service_pool.cc:225] Handling call
0625 13:52:16.368179 (+4815321us) ts_tablet_manager.cc:1145] Deleting tablet 38a23b08a03b48eaa051f670eb7dac65
0625 13:52:16.368191 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.401524 (+ 33333us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.404653 (+ 3129us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.434115 (+ 29462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4782026}
I20250625 13:52:16.434332 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_69c35651e2b2452894ec73fae8eea1bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.434813 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 38a23b08a03b48eaa051f670eb7dac65 (table default.Table_91567a207fcd4540937c0e047ea86de5 [id=5fcd0958c61942289b4eb28c077ceec3]) successfully deleted
W20250625 13:52:16.434908 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69c35651e2b2452894ec73fae8eea1bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.421137 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 882152a3a5a2458d803ba7f75b31bc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:16.437417 1165 log.cc:1199] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ab7202a8761444a0a731993d890fcdeb
W20250625 13:52:16.437803 983 consensus_peers.cc:487] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 430c3bed360b4f32855a8adb99076608. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:16.437885 1165 ts_tablet_manager.cc:1939] T ab7202a8761444a0a731993d890fcdeb P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.438988 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1017) took 4914 ms. Trace:
I20250625 13:52:16.438731 1171 tablet_bootstrap.cc:492] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.439141 1168 rpcz_store.cc:276] 0625 13:52:11.524535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:11.657287 (+132752us) service_pool.cc:225] Handling call
0625 13:52:16.435083 (+4777796us) ts_tablet_manager.cc:1145] Deleting tablet 4b61d4f7f0474aac9b46c532f6f403a0
0625 13:52:16.438888 (+ 3805us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.438969 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4777573}
I20250625 13:52:16.439399 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]) successfully deleted
W20250625 13:52:16.439630 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4b61d4f7f0474aac9b46c532f6f403a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b61d4f7f0474aac9b46c532f6f403a0 already in progress: creating tablet
I20250625 13:52:16.439954 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1030) took 4375 ms. Trace:
I20250625 13:52:16.440075 1168 rpcz_store.cc:276] 0625 13:52:12.064094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.064267 (+ 173us) service_pool.cc:225] Handling call
0625 13:52:16.439837 (+4375570us) ts_tablet_manager.cc:1145] Deleting tablet 913bc2f716c14de890bf1efdb0611dde
0625 13:52:16.439846 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.439937 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4375375}
W20250625 13:52:16.440527 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 913bc2f716c14de890bf1efdb0611dde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 913bc2f716c14de890bf1efdb0611dde already in progress: creating tablet
I20250625 13:52:16.440868 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1031) took 4376 ms. Trace:
I20250625 13:52:16.440976 1168 rpcz_store.cc:276] 0625 13:52:12.064221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.064498 (+ 277us) service_pool.cc:225] Handling call
0625 13:52:16.440770 (+4376272us) ts_tablet_manager.cc:1145] Deleting tablet 2593db823e974f1eb653470d5a23338f
0625 13:52:16.440778 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.440854 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4376156}
W20250625 13:52:16.441387 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2593db823e974f1eb653470d5a23338f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2593db823e974f1eb653470d5a23338f already in progress: creating tablet
I20250625 13:52:16.439728 1171 ts_tablet_manager.cc:1397] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.000s
I20250625 13:52:16.441743 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1047) took 4375 ms. Trace:
I20250625 13:52:16.442298 1168 rpcz_store.cc:276] 0625 13:52:12.066147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.066333 (+ 186us) service_pool.cc:225] Handling call
0625 13:52:16.441644 (+4375311us) ts_tablet_manager.cc:1145] Deleting tablet bdc5395c1de349b9b20d28625b8751f7
0625 13:52:16.441653 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.441727 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4375137}
W20250625 13:52:16.442638 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d8ce12926e044ee95291a5e9168277a consensus state has no leader
W20250625 13:52:16.442824 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bdc5395c1de349b9b20d28625b8751f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bdc5395c1de349b9b20d28625b8751f7 already in progress: creating tablet
I20250625 13:52:16.443248 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1046) took 4377 ms. Trace:
I20250625 13:52:16.443361 1168 rpcz_store.cc:276] 0625 13:52:12.066048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.082442 (+ 16394us) service_pool.cc:225] Handling call
0625 13:52:16.443128 (+4360686us) ts_tablet_manager.cc:1145] Deleting tablet 9302ac10a29c4b0c82268eb767615989
0625 13:52:16.443140 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.443230 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4360501}
W20250625 13:52:16.443769 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9302ac10a29c4b0c82268eb767615989 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9302ac10a29c4b0c82268eb767615989 already in progress: creating tablet
W20250625 13:52:16.443955 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 23fa1b7e7c13444fb8ee07807dabd813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23fa1b7e7c13444fb8ee07807dabd813 already in progress: creating tablet
I20250625 13:52:16.444118 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1043) took 4378 ms. Trace:
I20250625 13:52:16.444182 1168 rpcz_store.cc:276] 0625 13:52:12.065725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.087360 (+ 21635us) service_pool.cc:225] Handling call
0625 13:52:16.444007 (+4356647us) ts_tablet_manager.cc:1145] Deleting tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7
0625 13:52:16.444014 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.444103 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4356434}
W20250625 13:52:16.444504 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e3fc3242dd643be9dfe4fd6a7f3c9a7 already in progress: creating tablet
I20250625 13:52:16.444911 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1042) took 4379 ms. Trace:
I20250625 13:52:16.444999 1168 rpcz_store.cc:276] 0625 13:52:12.065651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.087612 (+ 21961us) service_pool.cc:225] Handling call
0625 13:52:16.444796 (+4357184us) ts_tablet_manager.cc:1145] Deleting tablet 54ea4c4d7756433d85fe9b816b4816ae
0625 13:52:16.444807 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.444893 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4357022}
W20250625 13:52:16.445402 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 54ea4c4d7756433d85fe9b816b4816ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54ea4c4d7756433d85fe9b816b4816ae already in progress: creating tablet
I20250625 13:52:16.445750 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1041) took 4380 ms. Trace:
I20250625 13:52:16.445842 1168 rpcz_store.cc:276] 0625 13:52:12.065569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.087800 (+ 22231us) service_pool.cc:225] Handling call
0625 13:52:16.445647 (+4357847us) ts_tablet_manager.cc:1145] Deleting tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604
0625 13:52:16.445657 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.445728 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4357679}
W20250625 13:52:16.446254 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc5a9cdb6804fe58fc3de1e7ba0d604 already in progress: creating tablet
I20250625 13:52:16.447137 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1032) took 4382 ms. Trace:
I20250625 13:52:16.447399 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 5e280bd329764924a6fd42c591888ed0 (table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]). Expected version 1 got 0
I20250625 13:52:16.447294 1168 rpcz_store.cc:276] 0625 13:52:12.064799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.095488 (+ 30689us) service_pool.cc:225] Handling call
0625 13:52:16.446933 (+4351445us) ts_tablet_manager.cc:1145] Deleting tablet c9d010744e88406a828c6ba7b76eda3e
0625 13:52:16.446945 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.447091 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4351217}
W20250625 13:52:16.447758 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c9d010744e88406a828c6ba7b76eda3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9d010744e88406a828c6ba7b76eda3e already in progress: creating tablet
I20250625 13:52:16.448323 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1062) took 4152 ms. Trace:
I20250625 13:52:16.448448 1168 rpcz_store.cc:276] 0625 13:52:12.296864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.296917 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:16.447909 (+4150992us) ts_tablet_manager.cc:1145] Deleting tablet ea383a1ceddc461fb2531dbca714029e
0625 13:52:16.448078 (+ 169us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.448279 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4150797}
I20250625 13:52:16.448616 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1172) took 2314 ms. Trace:
I20250625 13:52:16.448697 2254 rpcz_store.cc:276] 0625 13:52:14.133848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.133899 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:16.448472 (+2314573us) ts_tablet_manager.cc:1145] Deleting tablet 1f47260bf0ad41fdbfbad50380a12547
0625 13:52:16.448483 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.448595 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2314402}
W20250625 13:52:16.448874 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet ea383a1ceddc461fb2531dbca714029e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea383a1ceddc461fb2531dbca714029e already in progress: creating tablet
W20250625 13:52:16.449127 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1f47260bf0ad41fdbfbad50380a12547 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f47260bf0ad41fdbfbad50380a12547 already in progress: creating tablet
I20250625 13:52:16.449352 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c7173c8df6c647dbaf9eb9e4cc44e29d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.449517 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1063) took 4152 ms. Trace:
I20250625 13:52:16.449689 1168 rpcz_store.cc:276] 0625 13:52:12.297010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.298331 (+ 1321us) service_pool.cc:225] Handling call
0625 13:52:16.449383 (+4151052us) ts_tablet_manager.cc:1145] Deleting tablet c27caba6c4d14d51b10bf04f131c6a3c
0625 13:52:16.449391 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.449500 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4150866}
W20250625 13:52:16.449878 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7173c8df6c647dbaf9eb9e4cc44e29d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:16.450191 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet c27caba6c4d14d51b10bf04f131c6a3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c27caba6c4d14d51b10bf04f131c6a3c already in progress: creating tablet
I20250625 13:52:16.450587 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1058) took 4155 ms. Trace:
I20250625 13:52:16.450701 1168 rpcz_store.cc:276] 0625 13:52:12.295350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.295408 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:16.450441 (+4155033us) ts_tablet_manager.cc:1145] Deleting tablet 285912ccbf5a4e5bab53c48d1c024bb8
0625 13:52:16.450451 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.450573 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4146327}
W20250625 13:52:16.451134 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 285912ccbf5a4e5bab53c48d1c024bb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 285912ccbf5a4e5bab53c48d1c024bb8 already in progress: creating tablet
I20250625 13:52:16.451875 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7ed44586fa6f4aefaf28dbf32f10ed2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.452346 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ed44586fa6f4aefaf28dbf32f10ed2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.453212 1166 tablet_bootstrap.cc:492] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.453346 1166 ts_tablet_manager.cc:1397] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.094s user 0.007s sys 0.003s
W20250625 13:52:16.453864 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 882152a3a5a2458d803ba7f75b31bc30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 882152a3a5a2458d803ba7f75b31bc30 already in progress: creating tablet
I20250625 13:52:16.454289 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1061) took 4158 ms. Trace:
I20250625 13:52:16.454900 1166 raft_consensus.cc:357] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.390561 1010 rpcz_store.cc:276] 0625 13:52:15.297668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.297736 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:16.390416 (+1092680us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:52:16.420332 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 882152a3a5a2458d803ba7f75b31bc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.454983 1168 rpcz_store.cc:276] 0625 13:52:12.295763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.303506 (+ 7743us) service_pool.cc:225] Handling call
0625 13:52:16.453691 (+4150185us) ts_tablet_manager.cc:1145] Deleting tablet 6110bcce59ea452cbc6e9f464483bde9
0625 13:52:16.453703 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.454042 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4124898}
W20250625 13:52:16.456290 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6110bcce59ea452cbc6e9f464483bde9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6110bcce59ea452cbc6e9f464483bde9 already in progress: creating tablet
I20250625 13:52:16.457877 1166 raft_consensus.cc:383] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.457530 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1059) took 4162 ms. Trace:
I20250625 13:52:16.458127 1166 raft_consensus.cc:738] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.458734 1166 consensus_queue.cc:260] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.458237 1168 rpcz_store.cc:276] 0625 13:52:12.295486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.303694 (+ 8208us) service_pool.cc:225] Handling call
0625 13:52:16.456662 (+4152968us) ts_tablet_manager.cc:1145] Deleting tablet 4111b95923ed4d73b7eca60b964d9494
0625 13:52:16.456671 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.457515 (+ 844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4127724}
W20250625 13:52:16.459728 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 4111b95923ed4d73b7eca60b964d9494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4111b95923ed4d73b7eca60b964d9494 already in progress: creating tablet
I20250625 13:52:16.459846 1171 raft_consensus.cc:357] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.460873 1171 raft_consensus.cc:383] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.461020 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1076) took 3993 ms. Trace:
I20250625 13:52:16.461463 1171 raft_consensus.cc:738] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.461727 1168 rpcz_store.cc:276] 0625 13:52:12.467755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.468259 (+ 504us) service_pool.cc:225] Handling call
0625 13:52:16.460193 (+3991934us) ts_tablet_manager.cc:1145] Deleting tablet b70e7607709e4be395f1ed8d01d7d169
0625 13:52:16.460201 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.461007 (+ 806us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3991738}
W20250625 13:52:16.463290 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet b70e7607709e4be395f1ed8d01d7d169 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b70e7607709e4be395f1ed8d01d7d169 already in progress: creating tablet
I20250625 13:52:16.463227 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f2a8252e5ba44d768968c5df61d58d21" } new_table_name: "default.Table_930049e3536b44a1975c5f98bff68581" modify_external_catalogs: true
I20250625 13:52:16.463582 1171 consensus_queue.cc:260] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.465147 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1077) took 3996 ms. Trace:
I20250625 13:52:16.466154 1168 rpcz_store.cc:276] 0625 13:52:12.468769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.468813 (+ 44us) service_pool.cc:225] Handling call
0625 13:52:16.464227 (+3995414us) ts_tablet_manager.cc:1145] Deleting tablet 05364e23e0ec4cffb92a8df2d38ca4b9
0625 13:52:16.464236 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.465130 (+ 894us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3995039}
W20250625 13:52:16.467694 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 05364e23e0ec4cffb92a8df2d38ca4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05364e23e0ec4cffb92a8df2d38ca4b9 already in progress: creating tablet
I20250625 13:52:16.469599 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e38a85c299f04d60a3c465571fd33971
I20250625 13:52:16.470016 593 catalog_manager.cc:6652] ReplaceTablet: tablet e38a85c299f04d60a3c465571fd33971 deleted and replaced by tablet 654b1213646e42eeab8b97212aeb737e
I20250625 13:52:16.469866 1171 ts_tablet_manager.cc:1428] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250625 13:52:16.469729 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1078) took 4000 ms. Trace:
I20250625 13:52:16.470773 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e38a85c299f04d60a3c465571fd33971 with delete_type TABLET_DATA_DELETED (replaced by tablet 654b1213646e42eeab8b97212aeb737e) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:16.470981 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e38a85c299f04d60a3c465571fd33971 with delete_type TABLET_DATA_DELETED (replaced by tablet 654b1213646e42eeab8b97212aeb737e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.470461 1168 rpcz_store.cc:276] 0625 13:52:12.469127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.469167 (+ 40us) service_pool.cc:225] Handling call
0625 13:52:16.468682 (+3999515us) ts_tablet_manager.cc:1145] Deleting tablet 30b5b3444fd245bbb2ecd5889d956862
0625 13:52:16.468690 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.469711 (+ 1021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3999371}
I20250625 13:52:16.472357 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1173) took 2332 ms. Trace:
W20250625 13:52:16.472522 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 30b5b3444fd245bbb2ecd5889d956862 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30b5b3444fd245bbb2ecd5889d956862 already in progress: creating tablet
I20250625 13:52:16.472512 2254 rpcz_store.cc:276] 0625 13:52:14.140174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.140227 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:16.453622 (+2313395us) ts_tablet_manager.cc:1145] Deleting tablet 2b29392cfeb14f46a60fd0731f220127
0625 13:52:16.472226 (+ 18604us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.472335 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2313216}
I20250625 13:52:16.472889 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1174) took 2327 ms. Trace:
I20250625 13:52:16.472973 2254 rpcz_store.cc:276] 0625 13:52:14.145760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.146058 (+ 298us) service_pool.cc:225] Handling call
0625 13:52:16.472776 (+2326718us) ts_tablet_manager.cc:1145] Deleting tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc
0625 13:52:16.472788 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.472875 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2326533}
W20250625 13:52:16.473186 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2b29392cfeb14f46a60fd0731f220127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b29392cfeb14f46a60fd0731f220127 already in progress: creating tablet
I20250625 13:52:16.473359 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1175) took 2312 ms. Trace:
W20250625 13:52:16.473405 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc already in progress: creating tablet
I20250625 13:52:16.473528 2254 rpcz_store.cc:276] 0625 13:52:14.161179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.161231 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:16.473270 (+2312039us) ts_tablet_manager.cc:1145] Deleting tablet d7f68a0c4bb341a9a90cc2be0f78daca
0625 13:52:16.473278 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.473344 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2311247}
I20250625 13:52:16.473428 1171 tablet_bootstrap.cc:492] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:16.473934 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d7f68a0c4bb341a9a90cc2be0f78daca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7f68a0c4bb341a9a90cc2be0f78daca already in progress: creating tablet
I20250625 13:52:16.474299 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1176) took 2295 ms. Trace:
I20250625 13:52:16.474965 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1081) took 4004 ms. Trace:
I20250625 13:52:16.476274 2254 rpcz_store.cc:276] 0625 13:52:14.178399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.178664 (+ 265us) service_pool.cc:225] Handling call
0625 13:52:16.474188 (+2295524us) ts_tablet_manager.cc:1145] Deleting tablet fc9a79f066e84a7b87439ea4516f0287
0625 13:52:16.474199 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.474282 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2295324}
W20250625 13:52:16.476827 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet fc9a79f066e84a7b87439ea4516f0287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9a79f066e84a7b87439ea4516f0287 already in progress: creating tablet
I20250625 13:52:16.475811 1168 rpcz_store.cc:276] 0625 13:52:12.470607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.471425 (+ 818us) service_pool.cc:225] Handling call
0625 13:52:16.473952 (+4002527us) ts_tablet_manager.cc:1145] Deleting tablet a8c4f90b40c344168e8f689c43101a64
0625 13:52:16.473961 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.474949 (+ 988us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4002088}
I20250625 13:52:16.477231 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1177) took 2275 ms. Trace:
I20250625 13:52:16.477310 2254 rpcz_store.cc:276] 0625 13:52:14.202045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.202102 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:16.477117 (+2275015us) ts_tablet_manager.cc:1145] Deleting tablet fc9a79f066e84a7b87439ea4516f0287
0625 13:52:16.477128 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.477214 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2274816}
W20250625 13:52:16.477692 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet fc9a79f066e84a7b87439ea4516f0287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9a79f066e84a7b87439ea4516f0287 already in progress: creating tablet
I20250625 13:52:16.477797 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1186) took 2132 ms. Trace:
W20250625 13:52:16.477919 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a8c4f90b40c344168e8f689c43101a64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8c4f90b40c344168e8f689c43101a64 already in progress: creating tablet
I20250625 13:52:16.477887 2254 rpcz_store.cc:276] 0625 13:52:14.345088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.345138 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:16.477695 (+2132557us) ts_tablet_manager.cc:1145] Deleting tablet 3562e8b8731a4d76bb1a36189f957e07
0625 13:52:16.477706 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.477782 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2132166}
W20250625 13:52:16.478259 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 3562e8b8731a4d76bb1a36189f957e07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3562e8b8731a4d76bb1a36189f957e07 already in progress: creating tablet
I20250625 13:52:16.478637 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1187) took 2129 ms. Trace:
I20250625 13:52:16.478746 2254 rpcz_store.cc:276] 0625 13:52:14.349010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.349063 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:16.478557 (+2129494us) ts_tablet_manager.cc:1145] Deleting tablet 8f88091205b944b386094e50cf92c36b
0625 13:52:16.478563 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.478625 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129252}
W20250625 13:52:16.479158 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8f88091205b944b386094e50cf92c36b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f88091205b944b386094e50cf92c36b already in progress: creating tablet
I20250625 13:52:16.479456 2254 tablet_replica.cc:331] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.472128 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e38a85c299f04d60a3c465571fd33971 with delete_type TABLET_DATA_DELETED (replaced by tablet 654b1213646e42eeab8b97212aeb737e) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:16.481962 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e38a85c299f04d60a3c465571fd33971 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e38a85c299f04d60a3c465571fd33971 already in progress: creating tablet
I20250625 13:52:16.481529 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1080) took 4010 ms. Trace:
I20250625 13:52:16.482172 1166 ts_tablet_manager.cc:1428] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250625 13:52:16.482476 1166 tablet_bootstrap.cc:492] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.482069 1168 rpcz_store.cc:276] 0625 13:52:12.470506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.480091 (+ 9585us) service_pool.cc:225] Handling call
0625 13:52:16.480515 (+4000424us) ts_tablet_manager.cc:1145] Deleting tablet e438e46874f841d2ac972b953ddb4608
0625 13:52:16.480525 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.481278 (+ 753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3997718}
I20250625 13:52:16.484139 1166 tablet_bootstrap.cc:654] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:16.484247 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e438e46874f841d2ac972b953ddb4608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e438e46874f841d2ac972b953ddb4608 already in progress: creating tablet
I20250625 13:52:16.484881 2254 raft_consensus.cc:2241] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.485122 2254 raft_consensus.cc:2270] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.485730 2254 ts_tablet_manager.cc:1905] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.486263 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1083) took 4002 ms. Trace:
I20250625 13:52:16.487082 1168 rpcz_store.cc:276] 0625 13:52:12.483469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.483687 (+ 218us) service_pool.cc:225] Handling call
0625 13:52:16.485145 (+4001458us) ts_tablet_manager.cc:1145] Deleting tablet 34371795c8094e73ab4e5dbeedb7aa2f
0625 13:52:16.485154 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.485992 (+ 838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4000857}
W20250625 13:52:16.488788 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 34371795c8094e73ab4e5dbeedb7aa2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34371795c8094e73ab4e5dbeedb7aa2f already in progress: creating tablet
I20250625 13:52:16.491833 1168 tablet_replica.cc:331] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:16.492225 2411 raft_consensus.cc:491] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.492395 2411 raft_consensus.cc:513] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:16.492491 1168 raft_consensus.cc:2241] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.493105 2411 leader_election.cc:290] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.493816 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86af0ea6648045a1ac4ba903d20849b7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.494160 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86af0ea6648045a1ac4ba903d20849b7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:16.494977 985 leader_election.cc:304] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.494623 1168 raft_consensus.cc:2270] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.495390 2411 raft_consensus.cc:2747] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.495960 1171 tablet_bootstrap.cc:654] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.502223 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:16.503139 1168 ts_tablet_manager.cc:1905] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:16.505429 849 consensus_peers.cc:487] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d95a376304e464ab75e2e1e00e7a924. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:16.506198 1170 tablet_bootstrap.cc:492] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.506343 1170 ts_tablet_manager.cc:1397] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.005s
I20250625 13:52:16.507110 1170 raft_consensus.cc:357] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.507345 1170 raft_consensus.cc:383] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.507443 1170 raft_consensus.cc:738] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.507735 1170 consensus_queue.cc:260] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.512912 1170 ts_tablet_manager.cc:1428] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:16.513239 1170 tablet_bootstrap.cc:492] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.514669 2411 raft_consensus.cc:491] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.514789 2411 raft_consensus.cc:513] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.515844 2411 leader_election.cc:290] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.516407 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37e4bd4aaab143ffa65a668a952ae345" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.516582 915 raft_consensus.cc:2466] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
W20250625 13:52:16.516772 985 leader_election.cc:343] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 37e4bd4aaab143ffa65a668a952ae345
I20250625 13:52:16.517194 983 leader_election.cc:304] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:16.517673 2411 raft_consensus.cc:2802] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:16.517746 2411 raft_consensus.cc:491] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:16.517805 2411 raft_consensus.cc:3058] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:16.518471 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_652e0ff5dc3f4eb286b425ec0630c138" } new_table_name: "default.Table_a73ef917f48a41e28ee92a70c69e2f5b" modify_external_catalogs: true
I20250625 13:52:16.518863 1166 tablet_bootstrap.cc:492] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.519079 1166 ts_tablet_manager.cc:1397] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250625 13:52:16.520622 1166 raft_consensus.cc:357] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.521025 2411 raft_consensus.cc:513] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.521036 1166 raft_consensus.cc:383] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.521176 1166 raft_consensus.cc:738] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.521731 1166 consensus_queue.cc:260] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.522280 2411 leader_election.cc:290] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:16.522799 985 leader_election.cc:343] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 37e4bd4aaab143ffa65a668a952ae345
I20250625 13:52:16.523200 1166 ts_tablet_manager.cc:1428] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250625 13:52:16.523697 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:16.522370 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37e4bd4aaab143ffa65a668a952ae345" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:16.523998 915 raft_consensus.cc:3058] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:16.526589 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cebeffb6fa544997af268cc9579aee27 consensus state has no leader
I20250625 13:52:16.526736 915 raft_consensus.cc:2466] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:16.527177 983 leader_election.cc:304] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:16.527757 1166 tablet_bootstrap.cc:492] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.530257 2411 raft_consensus.cc:2802] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:16.540009 1171 tablet_bootstrap.cc:492] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.540287 1166 tablet_bootstrap.cc:654] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.540814 592 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:16.541356 1171 ts_tablet_manager.cc:1397] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20250625 13:52:16.542085 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_83b5cb8b7d284680a7e3a68341c7d2dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.543129 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83b5cb8b7d284680a7e3a68341c7d2dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.546689 1171 raft_consensus.cc:357] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.547609 2411 raft_consensus.cc:695] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:16.547493 1171 raft_consensus.cc:383] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.547875 1171 raft_consensus.cc:738] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.547978 2411 consensus_queue.cc:237] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.548873 1171 consensus_queue.cc:260] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.551883 1171 ts_tablet_manager.cc:1428] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250625 13:52:16.553087 1171 tablet_bootstrap.cc:492] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.560899 596 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:16.564452 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bdc29ce3c8744673b2c3c09c5f27acd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.564613 1166 tablet_bootstrap.cc:492] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.564918 1166 ts_tablet_manager.cc:1397] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.005s
W20250625 13:52:16.565320 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdc29ce3c8744673b2c3c09c5f27acd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.566468 1166 raft_consensus.cc:357] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.566823 1166 raft_consensus.cc:383] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.566967 1166 raft_consensus.cc:738] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.567418 1166 consensus_queue.cc:260] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.570250 1166 ts_tablet_manager.cc:1428] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 13:52:16.571432 1171 tablet_bootstrap.cc:654] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.572539 1166 tablet_bootstrap.cc:492] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.575549 1165 tablet_replica.cc:331] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:16.575738 1165 raft_consensus.cc:2241] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.575903 1165 raft_consensus.cc:2270] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.576431 1165 ts_tablet_manager.cc:1905] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.580433 2254 ts_tablet_manager.cc:1918] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:16.580592 2254 log.cc:1199] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/39ac79102eab46688e767faed6d6f114
I20250625 13:52:16.580958 2254 ts_tablet_manager.cc:1939] T 39ac79102eab46688e767faed6d6f114 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.581264 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1188) took 2229 ms. Trace:
I20250625 13:52:16.581447 1170 tablet_bootstrap.cc:654] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.581341 2254 rpcz_store.cc:276] 0625 13:52:14.351422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.354858 (+ 3436us) service_pool.cc:225] Handling call
0625 13:52:16.479417 (+2124559us) ts_tablet_manager.cc:1145] Deleting tablet 39ac79102eab46688e767faed6d6f114
0625 13:52:16.479425 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.530925 (+ 51500us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.580329 (+ 49404us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.581245 (+ 916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2124405}
W20250625 13:52:16.585675 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d8257f79a42247b48f2736a545362f44 (table default.Table_fd2fbb332353449080133a5788a93d74 [id=97d3f73d4cf64de1afc9c2397cdfa278], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d8257f79a42247b48f2736a545362f44 consensus state has no leader
I20250625 13:52:16.585839 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 39ac79102eab46688e767faed6d6f114 (table default.Table_f26987562c25452388d457eb3b600595 [id=62553d70f8ee470f944206ddc8cc2089]) successfully deleted
I20250625 13:52:16.587018 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1189) took 2229 ms. Trace:
W20250625 13:52:16.590314 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 60136.6us
I20250625 13:52:16.591352 2254 rpcz_store.cc:276] 0625 13:52:14.357427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.360869 (+ 3442us) service_pool.cc:225] Handling call
0625 13:52:16.586875 (+2226006us) ts_tablet_manager.cc:1145] Deleting tablet d0174345facc4216b6ddeb8f8c73790d
0625 13:52:16.586887 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.586996 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2225624}
W20250625 13:52:16.591934 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d0174345facc4216b6ddeb8f8c73790d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0174345facc4216b6ddeb8f8c73790d already in progress: creating tablet
I20250625 13:52:16.593173 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1198) took 2145 ms. Trace:
I20250625 13:52:16.593303 2254 rpcz_store.cc:276] 0625 13:52:14.448170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.457768 (+ 9598us) service_pool.cc:225] Handling call
0625 13:52:16.592240 (+2134472us) ts_tablet_manager.cc:1145] Deleting tablet 529c3bf8e72f46948cfb864445ff51f3
0625 13:52:16.593053 (+ 813us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.593152 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2134315}
W20250625 13:52:16.593786 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 529c3bf8e72f46948cfb864445ff51f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 529c3bf8e72f46948cfb864445ff51f3 already in progress: creating tablet
I20250625 13:52:16.594431 1165 ts_tablet_manager.cc:1918] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.595178 2254 tablet_replica.cc:331] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.595451 2254 raft_consensus.cc:2241] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.595693 2254 raft_consensus.cc:2270] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.596302 2254 ts_tablet_manager.cc:1905] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:16.598719 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 107998us
I20250625 13:52:16.601002 1165 log.cc:1199] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/32a57268a18f4f00a39f8360f9c90737
I20250625 13:52:16.601536 1165 ts_tablet_manager.cc:1939] T 32a57268a18f4f00a39f8360f9c90737 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:16.602460 2254 ts_tablet_manager.cc:1918] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250625 13:52:16.603765 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 32a57268a18f4f00a39f8360f9c90737 (table default.Table_61827a08a660423a91ddac5834f61382 [id=41c5c48c85234a85aa9ab76ba3f4c5c8]) successfully deleted
I20250625 13:52:16.607141 1168 ts_tablet_manager.cc:1918] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.607816 2398 raft_consensus.cc:491] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.607957 2398 raft_consensus.cc:513] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.608601 2398 leader_election.cc:290] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.608404 1168 log.cc:1199] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/490c8902efeb46c99543747a8787c1c7
I20250625 13:52:16.609653 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_876b0fe22f3147528bd9aedba430dc9f" } modify_external_catalogs: true
I20250625 13:52:16.609819 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_876b0fe22f3147528bd9aedba430dc9f" } modify_external_catalogs: true
I20250625 13:52:16.609886 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "669de0b7c93a4efea8623dab2e462c8d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:16.610502 711 leader_election.cc:343] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.611958 1168 ts_tablet_manager.cc:1939] T 490c8902efeb46c99543747a8787c1c7 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.616572 1166 tablet_bootstrap.cc:654] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.619894 2204 raft_consensus.cc:491] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.620033 2204 raft_consensus.cc:513] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.619925 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1084) took 4092 ms. Trace:
I20250625 13:52:16.620723 2204 leader_election.cc:290] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.621709 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:16.621908 775 raft_consensus.cc:2466] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:16.622320 851 leader_election.cc:304] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:16.620218 1168 rpcz_store.cc:276] 0625 13:52:12.527084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.527164 (+ 80us) service_pool.cc:225] Handling call
0625 13:52:16.490027 (+3962863us) ts_tablet_manager.cc:1145] Deleting tablet 490c8902efeb46c99543747a8787c1c7
0625 13:52:16.491756 (+ 1729us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.571894 (+ 80138us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.606590 (+ 34696us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.619674 (+ 13084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3962159}
I20250625 13:52:16.622982 1166 tablet_bootstrap.cc:492] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.623108 1166 ts_tablet_manager.cc:1397] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20250625 13:52:16.623631 2204 raft_consensus.cc:2802] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:16.623739 2204 raft_consensus.cc:491] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:16.623746 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 490c8902efeb46c99543747a8787c1c7 (table default.Table_2a8634c60e08492faffd30a254d0dc11 [id=a0e4a71b13d94a69affc3cc9e0cb5066]) successfully deleted
I20250625 13:52:16.623814 2204 raft_consensus.cc:3058] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:16.623873 1166 raft_consensus.cc:357] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.624022 1166 raft_consensus.cc:383] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.624075 1166 raft_consensus.cc:738] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.609907 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "669de0b7c93a4efea8623dab2e462c8d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.625626 2398 raft_consensus.cc:491] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.625757 2398 raft_consensus.cc:513] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:16.625821 711 leader_election.cc:343] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.625926 711 leader_election.cc:304] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:16.612757 985 consensus_peers.cc:487] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 430c3bed360b4f32855a8adb99076608. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:16.626236 2398 leader_election.cc:290] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.626355 2398 raft_consensus.cc:2747] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.616784 1170 tablet_bootstrap.cc:492] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.621519 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.626785 1170 ts_tablet_manager.cc:1397] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.114s user 0.008s sys 0.008s
I20250625 13:52:16.626881 2204 raft_consensus.cc:513] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.627467 1170 raft_consensus.cc:357] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.627635 1170 raft_consensus.cc:383] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.627686 1170 raft_consensus.cc:738] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.627842 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:16.627882 1170 consensus_queue.cc:260] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.627977 775 raft_consensus.cc:3058] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:16.628331 849 leader_election.cc:343] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.628861 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:16.630501 775 raft_consensus.cc:2466] T 423ca098138c476f861a19b44554fd24 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:16.630935 851 leader_election.cc:304] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:16.631065 849 leader_election.cc:343] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.631189 2271 raft_consensus.cc:2802] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:16.631299 2271 raft_consensus.cc:695] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:16.631553 2271 consensus_queue.cc:237] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.631834 1170 ts_tablet_manager.cc:1428] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 13:52:16.632091 1170 tablet_bootstrap.cc:492] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.634236 2204 leader_election.cc:290] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.635637 1171 tablet_bootstrap.cc:492] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.635772 1171 ts_tablet_manager.cc:1397] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.000s
I20250625 13:52:16.636446 1171 raft_consensus.cc:357] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.636595 1171 raft_consensus.cc:383] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.636641 1171 raft_consensus.cc:738] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.636804 1171 consensus_queue.cc:260] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.637315 1171 ts_tablet_manager.cc:1428] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 13:52:16.637547 1171 tablet_bootstrap.cc:492] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.639222 1171 tablet_bootstrap.cc:654] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.639981 2254 log.cc:1199] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/3f56a8a6cb964d3a979e2f1d944eeddc
I20250625 13:52:16.640333 2254 ts_tablet_manager.cc:1939] T 3f56a8a6cb964d3a979e2f1d944eeddc P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.640628 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1202) took 2081 ms. Trace:
I20250625 13:52:16.640704 2254 rpcz_store.cc:276] 0625 13:52:14.558977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.580751 (+ 21774us) service_pool.cc:225] Handling call
0625 13:52:16.594055 (+2013304us) ts_tablet_manager.cc:1145] Deleting tablet 3f56a8a6cb964d3a979e2f1d944eeddc
0625 13:52:16.595143 (+ 1088us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.599358 (+ 4215us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.602376 (+ 3018us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.640616 (+ 38240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2013091}
I20250625 13:52:16.641075 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1203) took 2032 ms. Trace:
I20250625 13:52:16.641132 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_2c311e79aef2487d8524720a4cd0677c [id=71729b0c8f7046bb8e80bb96f9f2ba71]) successfully deleted
I20250625 13:52:16.641160 2254 rpcz_store.cc:276] 0625 13:52:14.609101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.609159 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:16.640966 (+2031807us) ts_tablet_manager.cc:1145] Deleting tablet bb9fcf91fa304a359213a3e5267a9cac
0625 13:52:16.640974 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.641059 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2031573}
I20250625 13:52:16.641479 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1205) took 2028 ms. Trace:
W20250625 13:52:16.641568 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bb9fcf91fa304a359213a3e5267a9cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb9fcf91fa304a359213a3e5267a9cac already in progress: creating tablet
I20250625 13:52:16.641554 2254 rpcz_store.cc:276] 0625 13:52:14.613230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.613290 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:16.641390 (+2028100us) ts_tablet_manager.cc:1145] Deleting tablet 0346ed4f8d3f46aa84cbabbcadcebf23
0625 13:52:16.641398 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.641465 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027845}
I20250625 13:52:16.641858 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1204) took 2030 ms. Trace:
W20250625 13:52:16.641911 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0346ed4f8d3f46aa84cbabbcadcebf23 already in progress: creating tablet
I20250625 13:52:16.641935 2254 rpcz_store.cc:276] 0625 13:52:14.611503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.614102 (+ 2599us) service_pool.cc:225] Handling call
0625 13:52:16.641768 (+2027666us) ts_tablet_manager.cc:1145] Deleting tablet 83699efa521a44f092cb738002aa58e4
0625 13:52:16.641776 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.641844 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027424}
I20250625 13:52:16.642204 2254 tablet_replica.cc:331] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.642372 2254 raft_consensus.cc:2241] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.642544 2254 raft_consensus.cc:2270] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.643110 2254 ts_tablet_manager.cc:1905] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.644465 1170 tablet_bootstrap.cc:654] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.623629 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1085) took 4089 ms. Trace:
I20250625 13:52:16.646131 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ade61b1fec8c4d09a698917fe4a50bba" } modify_external_catalogs: true
I20250625 13:52:16.646304 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ade61b1fec8c4d09a698917fe4a50bba" } modify_external_catalogs: true
I20250625 13:52:16.646570 1168 rpcz_store.cc:276] 0625 13:52:12.534103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.534161 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:16.623023 (+4088862us) ts_tablet_manager.cc:1145] Deleting tablet e75bb35262df49c2bd68ad00b19733fd
0625 13:52:16.623034 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.623611 (+ 577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4088514}
W20250625 13:52:16.647156 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 83699efa521a44f092cb738002aa58e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83699efa521a44f092cb738002aa58e4 already in progress: creating tablet
W20250625 13:52:16.648092 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e75bb35262df49c2bd68ad00b19733fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e75bb35262df49c2bd68ad00b19733fd already in progress: creating tablet
I20250625 13:52:16.650002 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1086) took 4003 ms. Trace:
I20250625 13:52:16.650504 1168 rpcz_store.cc:276] 0625 13:52:12.646627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.646940 (+ 313us) service_pool.cc:225] Handling call
0625 13:52:16.648730 (+4001790us) ts_tablet_manager.cc:1145] Deleting tablet 6340de862fd04c12bb7f85557cfe53d2
0625 13:52:16.648882 (+ 152us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.649870 (+ 988us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3941749}
I20250625 13:52:16.651587 1166 consensus_queue.cc:260] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.652216 1166 ts_tablet_manager.cc:1428] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
W20250625 13:52:16.652273 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6340de862fd04c12bb7f85557cfe53d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6340de862fd04c12bb7f85557cfe53d2 already in progress: creating tablet
I20250625 13:52:16.654314 1166 tablet_bootstrap.cc:492] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.654333 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1089) took 3997 ms. Trace:
I20250625 13:52:16.656042 1166 tablet_bootstrap.cc:654] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.655207 1168 rpcz_store.cc:276] 0625 13:52:12.656781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.657670 (+ 889us) service_pool.cc:225] Handling call
0625 13:52:16.653310 (+3995640us) ts_tablet_manager.cc:1145] Deleting tablet f3c75deb64c74f0282efeed827659ec2
0625 13:52:16.653624 (+ 314us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.654318 (+ 694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3946316}
W20250625 13:52:16.656795 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet f3c75deb64c74f0282efeed827659ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3c75deb64c74f0282efeed827659ec2 already in progress: creating tablet
I20250625 13:52:16.661640 2398 raft_consensus.cc:491] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.661762 2398 raft_consensus.cc:513] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.662232 2398 leader_election.cc:290] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.664805 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ed893da9dad747c7869db417fd1c55e6" } modify_external_catalogs: true
I20250625 13:52:16.664963 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ed893da9dad747c7869db417fd1c55e6" } modify_external_catalogs: true
I20250625 13:52:16.665953 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ef50a3624b4c79acc803ff0e2ed364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.666074 2472 raft_consensus.cc:491] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.666190 2472 raft_consensus.cc:513] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.666806 2472 leader_election.cc:290] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.667543 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.669883 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c49ef682cbb46e1ae5c3b91e4e2d9c9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:16.670522 985 leader_election.cc:304] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.670892 2472 raft_consensus.cc:2747] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.679180 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_08c66d2c51c4482f905a1026a08ab47f" } modify_external_catalogs: true
I20250625 13:52:16.679414 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_08c66d2c51c4482f905a1026a08ab47f" } modify_external_catalogs: true
W20250625 13:52:16.679502 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 53684.6us
I20250625 13:52:16.680121 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.680994 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:16.682229 851 consensus_peers.cc:487] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab7202a8761444a0a731993d890fcdeb. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:16.683466 983 consensus_peers.cc:487] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 490c8902efeb46c99543747a8787c1c7. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:16.683624 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2b08fba37f4a38880c018ec2d869c8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.661203 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1087) took 4006 ms. Trace:
I20250625 13:52:16.684361 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2b08fba37f4a38880c018ec2d869c8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:16.686564 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53e9c9b3dbbb4b5bb29bfe78a0dbe443 has no consensus state
W20250625 13:52:16.690228 711 leader_election.cc:343] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:16.690506 711 leader_election.cc:343] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:16.690699 711 leader_election.cc:343] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.690824 711 leader_election.cc:304] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:16.691274 711 leader_election.cc:343] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.684614 1168 rpcz_store.cc:276] 0625 13:52:12.655214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.657850 (+ 2636us) service_pool.cc:225] Handling call
0625 13:52:16.659725 (+4001875us) ts_tablet_manager.cc:1145] Deleting tablet a85fa343e30e4e838ab81cd775c298ba
0625 13:52:16.659736 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.661185 (+ 1449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3952361}
I20250625 13:52:16.691440 711 leader_election.cc:304] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.691686 2398 raft_consensus.cc:2747] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.691859 2398 raft_consensus.cc:2747] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:16.692406 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a85fa343e30e4e838ab81cd775c298ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a85fa343e30e4e838ab81cd775c298ba already in progress: creating tablet
I20250625 13:52:16.694113 2398 raft_consensus.cc:491] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.694247 2398 raft_consensus.cc:513] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.694890 2398 leader_election.cc:290] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.695583 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.695969 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:16.696381 711 leader_election.cc:343] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:16.696636 711 leader_election.cc:343] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.696779 711 leader_election.cc:304] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.697024 2398 raft_consensus.cc:2747] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.699398 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1091) took 4034 ms. Trace:
I20250625 13:52:16.699594 1168 rpcz_store.cc:276] 0625 13:52:12.665308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.665386 (+ 78us) service_pool.cc:225] Handling call
0625 13:52:16.692959 (+4027573us) ts_tablet_manager.cc:1145] Deleting tablet ef1f9c03b0ef455cad05b3fe59bdd67c
0625 13:52:16.699215 (+ 6256us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.699372 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3985740}
I20250625 13:52:16.700656 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1088) took 4045 ms. Trace:
I20250625 13:52:16.700819 1168 rpcz_store.cc:276] 0625 13:52:12.655375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.661634 (+ 6259us) service_pool.cc:225] Handling call
0625 13:52:16.700545 (+4038911us) ts_tablet_manager.cc:1145] Deleting tablet e597f8cd8e5a4ec6be8db57f65c53c65
0625 13:52:16.700555 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.700636 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3992577}
W20250625 13:52:16.701288 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet e597f8cd8e5a4ec6be8db57f65c53c65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e597f8cd8e5a4ec6be8db57f65c53c65 already in progress: creating tablet
I20250625 13:52:16.701694 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1090) took 4036 ms. Trace:
I20250625 13:52:16.701825 1168 rpcz_store.cc:276] 0625 13:52:12.665166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.684977 (+ 19811us) service_pool.cc:225] Handling call
0625 13:52:16.701596 (+4016619us) ts_tablet_manager.cc:1145] Deleting tablet bd036ec08f774f5393067ce20bb07610
0625 13:52:16.701605 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.701680 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3993560}
W20250625 13:52:16.702315 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bd036ec08f774f5393067ce20bb07610 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd036ec08f774f5393067ce20bb07610 already in progress: creating tablet
I20250625 13:52:16.702733 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1106) took 3763 ms. Trace:
I20250625 13:52:16.702929 1168 rpcz_store.cc:276] 0625 13:52:12.938890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:12.939126 (+ 236us) service_pool.cc:225] Handling call
0625 13:52:16.702595 (+3763469us) ts_tablet_manager.cc:1145] Deleting tablet 466717307dd04d1cbdac9f03229f6a3b
0625 13:52:16.702603 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.702718 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3749217}
W20250625 13:52:16.703567 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 466717307dd04d1cbdac9f03229f6a3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 466717307dd04d1cbdac9f03229f6a3b already in progress: creating tablet
I20250625 13:52:16.704274 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1122) took 3711 ms. Trace:
I20250625 13:52:16.704495 1168 rpcz_store.cc:276] 0625 13:52:12.992936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.048194 (+ 55258us) service_pool.cc:225] Handling call
0625 13:52:16.704058 (+3655864us) ts_tablet_manager.cc:1145] Deleting tablet 9df1f195befe4aed999b41f770547fde
0625 13:52:16.704071 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.704252 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3655613}
I20250625 13:52:16.705533 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a4622e5472934b05b366bfa81a23ddf4" } new_table_name: "default.Table_eb5680877f724ba893b7661d6684834f" modify_external_catalogs: true
W20250625 13:52:16.705727 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9df1f195befe4aed999b41f770547fde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9df1f195befe4aed999b41f770547fde already in progress: creating tablet
I20250625 13:52:16.707441 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1123) took 3708 ms. Trace:
I20250625 13:52:16.708387 2254 ts_tablet_manager.cc:1918] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.708144 1168 rpcz_store.cc:276] 0625 13:52:12.998736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.048430 (+ 49694us) service_pool.cc:225] Handling call
0625 13:52:16.706465 (+3658035us) ts_tablet_manager.cc:1145] Deleting tablet a1263945be894618a7f9946e7ca50efd
0625 13:52:16.706474 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.707130 (+ 656us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3657918}
W20250625 13:52:16.709794 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a1263945be894618a7f9946e7ca50efd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1263945be894618a7f9946e7ca50efd already in progress: creating tablet
I20250625 13:52:16.709934 1170 tablet_bootstrap.cc:492] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.711352 2254 log.cc:1199] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/024f6cb363594420b0d5480fa79d156b
I20250625 13:52:16.711328 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1124) took 3712 ms. Trace:
I20250625 13:52:16.711462 1170 ts_tablet_manager.cc:1397] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.079s user 0.021s sys 0.006s
W20250625 13:52:16.712365 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdb955e3410945f898639235cd25bad7 consensus state has no leader
I20250625 13:52:16.711830 1168 rpcz_store.cc:276] 0625 13:52:12.998870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.048549 (+ 49679us) service_pool.cc:225] Handling call
0625 13:52:16.710491 (+3661942us) ts_tablet_manager.cc:1145] Deleting tablet fd7a6587aa674a0eaaa107e2a2f673a3
0625 13:52:16.710499 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.711081 (+ 582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3661843}
I20250625 13:52:16.712533 1170 raft_consensus.cc:357] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.712680 1170 raft_consensus.cc:383] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.712726 1170 raft_consensus.cc:738] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.712922 1170 consensus_queue.cc:260] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:16.713094 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet fd7a6587aa674a0eaaa107e2a2f673a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd7a6587aa674a0eaaa107e2a2f673a3 already in progress: creating tablet
I20250625 13:52:16.713369 1170 ts_tablet_manager.cc:1428] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:16.713655 1170 tablet_bootstrap.cc:492] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.716056 1170 tablet_bootstrap.cc:654] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.717787 2254 ts_tablet_manager.cc:1939] T 024f6cb363594420b0d5480fa79d156b P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.718603 1168 tablet_replica.cc:331] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:16.720367 1168 raft_consensus.cc:2241] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.721660 1168 raft_consensus.cc:2270] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.723555 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bd752418b476484dbe546a31f63ec6d9" } modify_external_catalogs: true
I20250625 13:52:16.723759 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bd752418b476484dbe546a31f63ec6d9" } modify_external_catalogs: true
W20250625 13:52:16.720000 985 consensus_peers.cc:487] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 490c8902efeb46c99543747a8787c1c7. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:16.727435 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1206) took 2005 ms. Trace:
I20250625 13:52:16.729513 1168 ts_tablet_manager.cc:1905] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.732106 2254 rpcz_store.cc:276] 0625 13:52:14.722314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.722369 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:16.642155 (+1919786us) ts_tablet_manager.cc:1145] Deleting tablet 024f6cb363594420b0d5480fa79d156b
0625 13:52:16.642165 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.665488 (+ 23323us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.708276 (+ 42788us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.727416 (+ 19140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829805}
I20250625 13:52:16.732628 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 024f6cb363594420b0d5480fa79d156b (table default.Table_81c1b426fda24349805e3e5cf9fdd561 [id=3eae0d10bcb247e0a318d6ac81e8c5da]) successfully deleted
I20250625 13:52:16.732998 2254 tablet_replica.cc:331] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.733212 2254 raft_consensus.cc:2241] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.733403 2254 raft_consensus.cc:2270] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.733999 2254 ts_tablet_manager.cc:1905] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.734999 1170 tablet_bootstrap.cc:492] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.735671 1170 ts_tablet_manager.cc:1397] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.001s
I20250625 13:52:16.736395 1170 raft_consensus.cc:357] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.739154 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9925fd56bfad4215a806983cb204acd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.739729 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9925fd56bfad4215a806983cb204acd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.740082 1170 raft_consensus.cc:383] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.740152 1170 raft_consensus.cc:738] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.740384 1170 consensus_queue.cc:260] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.741482 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf7fab53cae49908d2952cdaa07943a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.742293 1170 ts_tablet_manager.cc:1428] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:16.742674 1170 tablet_bootstrap.cc:492] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.744527 1170 tablet_bootstrap.cc:654] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.748677 1171 tablet_bootstrap.cc:492] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.748818 1171 ts_tablet_manager.cc:1397] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.002s
I20250625 13:52:16.749529 1171 raft_consensus.cc:357] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.749696 1171 raft_consensus.cc:383] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.749748 1171 raft_consensus.cc:738] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.749927 1171 consensus_queue.cc:260] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.750422 1171 ts_tablet_manager.cc:1428] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:16.751459 2472 raft_consensus.cc:491] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.751600 2472 raft_consensus.cc:513] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.752285 2472 leader_election.cc:290] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:16.753167 985 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 74884bf313724d678d3b5f5869f09f97
I20250625 13:52:16.754484 1171 tablet_bootstrap.cc:492] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.755780 2254 ts_tablet_manager.cc:1918] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.755887 2254 log.cc:1199] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/042359eacaa2451abc0fa6ed447a2905
I20250625 13:52:16.756167 2254 ts_tablet_manager.cc:1939] T 042359eacaa2451abc0fa6ed447a2905 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:16.756433 983 leader_election.cc:343] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 74884bf313724d678d3b5f5869f09f97
I20250625 13:52:16.756543 983 leader_election.cc:304] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.757081 2472 raft_consensus.cc:2747] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.760413 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_54c3accf07ed49259aa5cfdfcd715f72" } new_table_name: "default.Table_db9928537f0343b9b65d1109850804fc" modify_external_catalogs: true
W20250625 13:52:16.763890 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 078a05d5168a46698535a8d3fd08f634 has no consensus state
I20250625 13:52:16.764057 1171 tablet_bootstrap.cc:654] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.764292 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_0e4bb21a95674c7bb2e602b23226a395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.764937 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e4bb21a95674c7bb2e602b23226a395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.772123 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f49c0fc97d44b28acfd6cb493906edb from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:16.772872 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a7af3ed3af9f441c85af4605424c5f90" } new_table_name: "default.Table_70e8ca6bcdff497daaf2b4dcc9a8183e" modify_external_catalogs: true
I20250625 13:52:16.773538 2472 raft_consensus.cc:491] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.773691 2472 raft_consensus.cc:513] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.774367 2472 leader_election.cc:290] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.774914 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.774935 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e280bd329764924a6fd42c591888ed0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:16.775478 985 leader_election.cc:304] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.775784 2478 raft_consensus.cc:491] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.775966 2472 raft_consensus.cc:2747] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.775931 2478 raft_consensus.cc:513] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.776806 2478 leader_election.cc:290] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.776844 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9462da7574a46839763b1c388102e60" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:16.777369 711 leader_election.cc:343] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.777722 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9462da7574a46839763b1c388102e60" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:16.778170 711 leader_election.cc:343] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.778275 711 leader_election.cc:304] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.778502 2478 raft_consensus.cc:2747] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.780423 1170 tablet_bootstrap.cc:492] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.780781 1170 ts_tablet_manager.cc:1397] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.003s
I20250625 13:52:16.782476 1170 raft_consensus.cc:357] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.782861 1170 raft_consensus.cc:383] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.783020 1170 raft_consensus.cc:738] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.783474 1170 consensus_queue.cc:260] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.784288 1170 ts_tablet_manager.cc:1428] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:16.785177 1170 tablet_bootstrap.cc:492] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.756453 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1207) took 2014 ms. Trace:
I20250625 13:52:16.786548 2254 rpcz_store.cc:276] 0625 13:52:14.741821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.741874 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:16.732948 (+1991074us) ts_tablet_manager.cc:1145] Deleting tablet 042359eacaa2451abc0fa6ed447a2905
0625 13:52:16.732958 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.752586 (+ 19628us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.755692 (+ 3106us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.756439 (+ 747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1920550}
I20250625 13:52:16.789532 1170 tablet_bootstrap.cc:654] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.789822 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 042359eacaa2451abc0fa6ed447a2905 (table default.Table_3c1061dac55a4e97ad139950a7c8f302 [id=426662edb72f4d1aa3e0aa1ee934e62d]) successfully deleted
I20250625 13:52:16.790395 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1209) took 2007 ms. Trace:
I20250625 13:52:16.796005 2254 rpcz_store.cc:276] 0625 13:52:14.783011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.783068 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:16.790269 (+2007201us) ts_tablet_manager.cc:1145] Deleting tablet 9710609da88d4b42993b3aef02416543
0625 13:52:16.790280 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.790380 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1977358}
I20250625 13:52:16.796207 1166 tablet_bootstrap.cc:492] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.796341 1166 ts_tablet_manager.cc:1397] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.142s user 0.008s sys 0.009s
W20250625 13:52:16.796602 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9710609da88d4b42993b3aef02416543 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9710609da88d4b42993b3aef02416543 already in progress: creating tablet
I20250625 13:52:16.797206 1166 raft_consensus.cc:357] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.798245 1168 ts_tablet_manager.cc:1918] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.798527 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1213) took 2015 ms. Trace:
I20250625 13:52:16.798647 2254 rpcz_store.cc:276] 0625 13:52:14.783470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.783962 (+ 492us) service_pool.cc:225] Handling call
0625 13:52:16.796954 (+2012992us) ts_tablet_manager.cc:1145] Deleting tablet bbd3200bd1384187978aeb7089ec4e03
0625 13:52:16.798380 (+ 1426us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.798503 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1983581}
W20250625 13:52:16.799166 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bbd3200bd1384187978aeb7089ec4e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbd3200bd1384187978aeb7089ec4e03 already in progress: creating tablet
I20250625 13:52:16.799566 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1222) took 1926 ms. Trace:
I20250625 13:52:16.799679 2254 rpcz_store.cc:276] 0625 13:52:14.873003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.873309 (+ 306us) service_pool.cc:225] Handling call
0625 13:52:16.799460 (+1926151us) ts_tablet_manager.cc:1145] Deleting tablet d2e8f4e3b45a4065b74485157f31247d
0625 13:52:16.799470 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.799552 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1925928}
W20250625 13:52:16.800168 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d2e8f4e3b45a4065b74485157f31247d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2e8f4e3b45a4065b74485157f31247d already in progress: creating tablet
I20250625 13:52:16.800617 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1223) took 1927 ms. Trace:
I20250625 13:52:16.800750 2254 rpcz_store.cc:276] 0625 13:52:14.873138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.873594 (+ 456us) service_pool.cc:225] Handling call
0625 13:52:16.800475 (+1926881us) ts_tablet_manager.cc:1145] Deleting tablet 29e223675c7842b383865a270b97ce58
0625 13:52:16.800484 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.800597 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1926705}
I20250625 13:52:16.799516 2271 raft_consensus.cc:491] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.801040 1168 log.cc:1199] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/dfaba56442c0498da95e6fad126ab86d
I20250625 13:52:16.801077 2271 raft_consensus.cc:513] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.801374 1168 ts_tablet_manager.cc:1939] T dfaba56442c0498da95e6fad126ab86d P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.801368 1166 raft_consensus.cc:383] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.801453 1166 raft_consensus.cc:738] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.801653 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1125) took 3802 ms. Trace:
I20250625 13:52:16.801712 1166 consensus_queue.cc:260] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.801723 1168 rpcz_store.cc:276] 0625 13:52:12.998959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.048657 (+ 49698us) service_pool.cc:225] Handling call
0625 13:52:16.718027 (+3669370us) ts_tablet_manager.cc:1145] Deleting tablet dfaba56442c0498da95e6fad126ab86d
0625 13:52:16.718040 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.754051 (+ 36011us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.798109 (+ 44058us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.801638 (+ 3529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3669048}
I20250625 13:52:16.802093 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet dfaba56442c0498da95e6fad126ab86d (table default.Table_f4de7823c10e40e1bdd42b0e0559f7bf [id=7270b5aa9a0148178683a229db5b4b7a]) successfully deleted
I20250625 13:52:16.802227 1166 ts_tablet_manager.cc:1428] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20250625 13:52:16.802460 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 29e223675c7842b383865a270b97ce58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29e223675c7842b383865a270b97ce58 already in progress: creating tablet
I20250625 13:52:16.802410 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dec08d20a09444f5b9a102a8712e36c7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.802520 1166 tablet_bootstrap.cc:492] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.802755 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1224) took 1929 ms. Trace:
I20250625 13:52:16.802824 2254 rpcz_store.cc:276] 0625 13:52:14.873261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.873772 (+ 511us) service_pool.cc:225] Handling call
0625 13:52:16.802664 (+1928892us) ts_tablet_manager.cc:1145] Deleting tablet 0bc136dc84de44b49abcdf4100a86137
0625 13:52:16.802672 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.802732 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1928781}
W20250625 13:52:16.803134 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 0bc136dc84de44b49abcdf4100a86137 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bc136dc84de44b49abcdf4100a86137 already in progress: creating tablet
I20250625 13:52:16.805912 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1226) took 1919 ms. Trace:
I20250625 13:52:16.805982 2254 rpcz_store.cc:276] 0625 13:52:14.886302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.886363 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:16.805820 (+1919457us) ts_tablet_manager.cc:1145] Deleting tablet 1b2914ce9b4c40a0bc00d0dd5ac72221
0625 13:52:16.805828 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.805897 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919264}
W20250625 13:52:16.806316 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 already in progress: creating tablet
I20250625 13:52:16.808547 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3859f399a7b84de99658380bd6c24ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:16.808476 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c4c8c1a0038e4cf6add4073a66492474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.808785 2254 tablet_replica.cc:331] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.808962 2254 raft_consensus.cc:2241] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
W20250625 13:52:16.808992 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4c8c1a0038e4cf6add4073a66492474 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.809232 2254 pending_rounds.cc:70] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:16.809275 2254 pending_rounds.cc:77] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520657130139648 op_type: NO_OP noop_request { }
I20250625 13:52:16.809353 2254 raft_consensus.cc:2887] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:16.809423 2254 raft_consensus.cc:2270] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.809975 2254 ts_tablet_manager.cc:1905] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.812196 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dec08d20a09444f5b9a102a8712e36c7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:52:16.812356 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0d4536a3a69448e9d9acca67393f0da consensus state has no leader
I20250625 13:52:16.812424 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1121) took 3819 ms. Trace:
I20250625 13:52:16.812438 775 raft_consensus.cc:2466] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:16.812505 1168 rpcz_store.cc:276] 0625 13:52:12.992772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.047256 (+ 54484us) service_pool.cc:225] Handling call
0625 13:52:16.812278 (+3765022us) ts_tablet_manager.cc:1145] Deleting tablet 909f4bb91e784b29acbbc9763e35ab03
0625 13:52:16.812293 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.812405 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3752708}
I20250625 13:52:16.812788 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6a20710d360b44daa77b9e6fb6063303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.812919 851 leader_election.cc:304] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:16.812937 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 909f4bb91e784b29acbbc9763e35ab03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 909f4bb91e784b29acbbc9763e35ab03 already in progress: creating tablet
W20250625 13:52:16.813061 849 leader_election.cc:343] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.813200 2204 raft_consensus.cc:2802] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:16.813270 2204 raft_consensus.cc:491] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:16.813328 2204 raft_consensus.cc:3058] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:16.813338 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a20710d360b44daa77b9e6fb6063303 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.801744 2271 leader_election.cc:290] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.814359 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1139) took 3265 ms. Trace:
I20250625 13:52:16.814425 1168 rpcz_store.cc:276] 0625 13:52:13.548410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.548469 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:16.814259 (+3265790us) ts_tablet_manager.cc:1145] Deleting tablet a2ebcb65c49746f3ba9aa905e657c107
0625 13:52:16.814269 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.814343 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3239627}
W20250625 13:52:16.814828 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a2ebcb65c49746f3ba9aa905e657c107 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ebcb65c49746f3ba9aa905e657c107 already in progress: creating tablet
I20250625 13:52:16.815171 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1137) took 3283 ms. Trace:
I20250625 13:52:16.815243 1168 rpcz_store.cc:276] 0625 13:52:13.531979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.554469 (+ 22490us) service_pool.cc:225] Handling call
0625 13:52:16.815049 (+3260580us) ts_tablet_manager.cc:1145] Deleting tablet 71170aaf26ed4f1b8c458d90db6e2765
0625 13:52:16.815059 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.815154 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3239477}
W20250625 13:52:16.815589 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 71170aaf26ed4f1b8c458d90db6e2765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71170aaf26ed4f1b8c458d90db6e2765 already in progress: creating tablet
W20250625 13:52:16.816103 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25c71f11222e48818ea94984a985d6cb consensus state has no leader
I20250625 13:52:16.816912 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1ec3d8755b0d4d0f86175e3f35c24662"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.817011 2204 raft_consensus.cc:513] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:16.817520 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ec3d8755b0d4d0f86175e3f35c24662 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.818151 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dec08d20a09444f5b9a102a8712e36c7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:16.818176 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dec08d20a09444f5b9a102a8712e36c7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:16.818325 775 raft_consensus.cc:3058] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:16.818773 849 leader_election.cc:343] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.821107 775 raft_consensus.cc:2466] T dec08d20a09444f5b9a102a8712e36c7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:16.821527 851 leader_election.cc:304] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:16.821969 2271 raft_consensus.cc:2802] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:16.822108 2271 raft_consensus.cc:695] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:16.822343 2271 consensus_queue.cc:237] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.822789 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1138) took 3274 ms. Trace:
I20250625 13:52:16.822882 1168 rpcz_store.cc:276] 0625 13:52:13.548296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.569626 (+ 21330us) service_pool.cc:225] Handling call
0625 13:52:16.822652 (+3253026us) ts_tablet_manager.cc:1145] Deleting tablet 40f7656d6e53489ab982ba96bc04771b
0625 13:52:16.822662 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.822769 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3246978}
W20250625 13:52:16.823351 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 40f7656d6e53489ab982ba96bc04771b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40f7656d6e53489ab982ba96bc04771b already in progress: creating tablet
I20250625 13:52:16.823501 2204 leader_election.cc:290] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:16.828563 1168 tablet_replica.cc:331] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:16.828871 1168 raft_consensus.cc:2241] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.830062 1168 raft_consensus.cc:2270] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.831718 1166 tablet_bootstrap.cc:654] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.834575 1168 ts_tablet_manager.cc:1905] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.839154 1166 tablet_bootstrap.cc:492] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.839385 1166 ts_tablet_manager.cc:1397] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.003s
I20250625 13:52:16.840176 1166 raft_consensus.cc:357] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.847409 1166 raft_consensus.cc:383] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.847563 1166 raft_consensus.cc:738] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.847812 1166 consensus_queue.cc:260] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.848788 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:16.851859 1166 ts_tablet_manager.cc:1428] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250625 13:52:16.852375 1166 tablet_bootstrap.cc:492] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.854449 1166 tablet_bootstrap.cc:654] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.858294 2478 raft_consensus.cc:491] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.858457 2478 raft_consensus.cc:513] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.859186 2478 leader_election.cc:290] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.860158 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249c5c2aa564027a2c594ca4c31e1c8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.860601 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249c5c2aa564027a2c594ca4c31e1c8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.861052 2254 ts_tablet_manager.cc:1918] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:16.861162 2254 log.cc:1199] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/430c3bed360b4f32855a8adb99076608
I20250625 13:52:16.861507 2254 ts_tablet_manager.cc:1939] T 430c3bed360b4f32855a8adb99076608 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:16.861644 711 leader_election.cc:343] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.861826 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1232) took 1863 ms. Trace:
W20250625 13:52:16.861847 711 leader_election.cc:343] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.861925 711 leader_election.cc:304] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.861903 2254 rpcz_store.cc:276] 0625 13:52:14.998272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.003361 (+ 5089us) service_pool.cc:225] Handling call
0625 13:52:16.808733 (+1805372us) ts_tablet_manager.cc:1145] Deleting tablet 430c3bed360b4f32855a8adb99076608
0625 13:52:16.808753 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.836407 (+ 27654us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.860941 (+ 24534us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.861812 (+ 871us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1805015}
I20250625 13:52:16.862178 1166 tablet_bootstrap.cc:492] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:16.862217 2254 tablet_replica.cc:331] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.862300 1166 ts_tablet_manager.cc:1397] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20250625 13:52:16.862397 2254 raft_consensus.cc:2241] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.862373 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 430c3bed360b4f32855a8adb99076608 (table default.Table_1099b7a2614540d3ac476e1bad30554c [id=85c2a3c9e7b84cff94e2b24b4bdeb0c3]) successfully deleted
I20250625 13:52:16.863974 1166 raft_consensus.cc:357] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.867036 1168 ts_tablet_manager.cc:1918] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.862584 2254 raft_consensus.cc:2270] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.862905 2478 raft_consensus.cc:2747] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.867583 1166 raft_consensus.cc:383] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.867681 1168 log.cc:1199] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/569f6cba69394f2fa005686056f19929
I20250625 13:52:16.867683 1166 raft_consensus.cc:738] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.868109 1168 ts_tablet_manager.cc:1939] T 569f6cba69394f2fa005686056f19929 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.868309 2254 ts_tablet_manager.cc:1905] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.868394 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_fe109a1bd53e481d9dc7f6626b5c3759"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.868535 1166 consensus_queue.cc:260] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.869020 1166 ts_tablet_manager.cc:1428] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:16.869362 1166 tablet_bootstrap.cc:492] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:16.869396 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1141) took 3303 ms. Trace:
W20250625 13:52:16.868978 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe109a1bd53e481d9dc7f6626b5c3759 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.870999 1166 tablet_bootstrap.cc:654] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.868337 1171 tablet_bootstrap.cc:492] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.880414 1171 ts_tablet_manager.cc:1397] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.008s
I20250625 13:52:16.881259 1171 raft_consensus.cc:357] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.881485 1171 raft_consensus.cc:383] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.881570 1171 raft_consensus.cc:738] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.881798 1171 consensus_queue.cc:260] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.882387 1171 ts_tablet_manager.cc:1428] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:16.882520 1168 rpcz_store.cc:276] 0625 13:52:13.566038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.571357 (+ 5319us) service_pool.cc:225] Handling call
0625 13:52:16.828484 (+3257127us) ts_tablet_manager.cc:1145] Deleting tablet 569f6cba69394f2fa005686056f19929
0625 13:52:16.828497 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.852717 (+ 24220us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.866933 (+ 14216us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.869377 (+ 2444us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252090}
I20250625 13:52:16.883839 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_39f734a42d3845478cc2f4f5fe79181f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.884375 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39f734a42d3845478cc2f4f5fe79181f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.885349 1170 tablet_bootstrap.cc:492] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:16.885646 1170 ts_tablet_manager.cc:1397] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.101s user 0.028s sys 0.010s
I20250625 13:52:16.885782 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 569f6cba69394f2fa005686056f19929 (table default.Table_06d8369e7a09451eba488006b53da953 [id=421fdfc88bfe40349eae76010c55e7ab]) successfully deleted
I20250625 13:52:16.885447 1171 tablet_bootstrap.cc:492] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.887883 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1140) took 3339 ms. Trace:
I20250625 13:52:16.888214 1168 rpcz_store.cc:276] 0625 13:52:13.548874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.569596 (+ 20722us) service_pool.cc:225] Handling call
0625 13:52:16.887748 (+3318152us) ts_tablet_manager.cc:1145] Deleting tablet 3713b85743d04649b27517412b3d9ea0
0625 13:52:16.887757 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.887866 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3308668}
I20250625 13:52:16.889632 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:16.888680 1170 raft_consensus.cc:357] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.890244 1170 raft_consensus.cc:383] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.890661 1170 raft_consensus.cc:738] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:16.891489 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3713b85743d04649b27517412b3d9ea0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3713b85743d04649b27517412b3d9ea0 already in progress: creating tablet
I20250625 13:52:16.891492 1170 consensus_queue.cc:260] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.894261 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_451b836c46da4b9db775fad108a69547"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.894809 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_451b836c46da4b9db775fad108a69547 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.896387 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:16.904834 1170 ts_tablet_manager.cc:1428] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20250625 13:52:16.888162 1171 tablet_bootstrap.cc:654] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.906831 1168 tablet_replica.cc:331] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:16.911506 1168 raft_consensus.cc:2241] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.911718 1168 raft_consensus.cc:2270] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.912338 1168 ts_tablet_manager.cc:1905] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.915845 2472 raft_consensus.cc:491] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.915975 2472 raft_consensus.cc:513] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.916529 2472 leader_election.cc:290] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.917686 2254 ts_tablet_manager.cc:1918] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.917863 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9514bbbda0ba4329a18b0a42e635c6e5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.923800 1171 tablet_bootstrap.cc:492] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.923972 1171 ts_tablet_manager.cc:1397] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250625 13:52:16.924850 1171 raft_consensus.cc:357] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.925048 1171 raft_consensus.cc:383] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.925135 1171 raft_consensus.cc:738] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.925390 1171 consensus_queue.cc:260] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.925908 1171 ts_tablet_manager.cc:1428] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:16.929399 1168 ts_tablet_manager.cc:1918] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:16.929641 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ebb1cd6aa381410a82cac38e7197353c" } new_table_name: "default.Table_add52e39ed4c414c9c9e930208ab5c10" modify_external_catalogs: true
I20250625 13:52:16.931207 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7096ca87d4754bec97389b249556e073" } new_table_name: "default.Table_1653afaa260d4f76bad5671a7169c56e" modify_external_catalogs: true
W20250625 13:52:16.918231 985 leader_election.cc:343] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9514bbbda0ba4329a18b0a42e635c6e5
I20250625 13:52:16.921154 1170 tablet_bootstrap.cc:492] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:16.928947 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 723246e1e6824faf9f9d39c3ea941fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:16.933396 983 leader_election.cc:343] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: Tablet not RUNNING: STOPPED
I20250625 13:52:16.933642 2254 log.cc:1199] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/6c5fc63828d9408b9cf774a966f43cf2
I20250625 13:52:16.933668 983 leader_election.cc:304] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.934056 2254 ts_tablet_manager.cc:1939] T 6c5fc63828d9408b9cf774a966f43cf2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:16.934458 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1233) took 1915 ms. Trace:
I20250625 13:52:16.936655 2254 rpcz_store.cc:276] 0625 13:52:15.019116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.019168 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:16.862174 (+1843006us) ts_tablet_manager.cc:1145] Deleting tablet 6c5fc63828d9408b9cf774a966f43cf2
0625 13:52:16.862183 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.872530 (+ 10347us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.917581 (+ 45051us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.934439 (+ 16858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1842780}
I20250625 13:52:16.935865 1170 tablet_bootstrap.cc:654] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:16.937141 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 6c5fc63828d9408b9cf774a966f43cf2 (table default.Table_de6ae2243e604e0a8094e43c5b76ca27 [id=6de15b35e99646eb8eba9c11eea579c2]) successfully deleted
I20250625 13:52:16.937577 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1238) took 1839 ms. Trace:
I20250625 13:52:16.937686 2254 rpcz_store.cc:276] 0625 13:52:15.098618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.102466 (+ 3848us) service_pool.cc:225] Handling call
0625 13:52:16.937437 (+1834971us) ts_tablet_manager.cc:1145] Deleting tablet 6e2d109e6fce407dad72a2469479b5ed
0625 13:52:16.937451 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.937560 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834782}
W20250625 13:52:16.938730 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6e2d109e6fce407dad72a2469479b5ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2d109e6fce407dad72a2469479b5ed already in progress: creating tablet
I20250625 13:52:16.939167 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1240) took 1735 ms. Trace:
I20250625 13:52:16.946321 2254 rpcz_store.cc:276] 0625 13:52:15.203725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.203789 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:16.939043 (+1735254us) ts_tablet_manager.cc:1145] Deleting tablet 17cf1172bcc74cca930824f9cda95f3b
0625 13:52:16.939055 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.939145 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1735039}
I20250625 13:52:16.946792 2478 raft_consensus.cc:491] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.946954 2478 raft_consensus.cc:513] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.947683 2478 leader_election.cc:290] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.948172 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_29f4b702a71c42d38dc12caed1b9e4b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:16.948352 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2c3757c0284aaca70ec317c05195d6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:16.948529 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 17cf1172bcc74cca930824f9cda95f3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17cf1172bcc74cca930824f9cda95f3b already in progress: creating tablet
W20250625 13:52:16.948793 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29f4b702a71c42d38dc12caed1b9e4b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:16.948869 711 leader_election.cc:343] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.948925 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1241) took 1651 ms. Trace:
I20250625 13:52:16.949011 2254 rpcz_store.cc:276] 0625 13:52:15.297231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.301184 (+ 3953us) service_pool.cc:225] Handling call
0625 13:52:16.948804 (+1647620us) ts_tablet_manager.cc:1145] Deleting tablet f47980e0c9ad487b8113f13458e175ec
0625 13:52:16.948818 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.948905 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1552462}
W20250625 13:52:16.949455 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet f47980e0c9ad487b8113f13458e175ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47980e0c9ad487b8113f13458e175ec already in progress: creating tablet
I20250625 13:52:16.951701 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1242) took 1636 ms. Trace:
I20250625 13:52:16.951848 2254 rpcz_store.cc:276] 0625 13:52:15.315545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.315605 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:16.951586 (+1635981us) ts_tablet_manager.cc:1145] Deleting tablet 9252dde349d24a8c99dfe2eda37c9f64
0625 13:52:16.951594 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.951680 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1553967}
I20250625 13:52:16.929754 1171 tablet_bootstrap.cc:492] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:16.950794 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2c3757c0284aaca70ec317c05195d6" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:16.954110 1171 tablet_bootstrap.cc:654] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:16.955148 711 leader_election.cc:343] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.955279 711 leader_election.cc:304] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:16.955493 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 106454us
I20250625 13:52:16.955577 2478 raft_consensus.cc:2747] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.955888 1168 log.cc:1199] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/9514bbbda0ba4329a18b0a42e635c6e5
I20250625 13:52:16.969249 1168 ts_tablet_manager.cc:1939] T 9514bbbda0ba4329a18b0a42e635c6e5 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:16.970296 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1143) took 3368 ms. Trace:
I20250625 13:52:16.970423 594 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
W20250625 13:52:16.970844 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9252dde349d24a8c99dfe2eda37c9f64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9252dde349d24a8c99dfe2eda37c9f64 already in progress: creating tablet
I20250625 13:52:16.971393 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1243) took 1587 ms. Trace:
I20250625 13:52:16.971539 2254 rpcz_store.cc:276] 0625 13:52:15.384237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.384290 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:16.971203 (+1586913us) ts_tablet_manager.cc:1145] Deleting tablet d0baddb123f0401a98662b4b483b9587
0625 13:52:16.971215 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.971368 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1572202}
W20250625 13:52:16.972106 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d0baddb123f0401a98662b4b483b9587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0baddb123f0401a98662b4b483b9587 already in progress: creating tablet
I20250625 13:52:16.972581 2254 tablet_replica.cc:331] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:16.979503 2254 raft_consensus.cc:2241] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:16.980157 1168 rpcz_store.cc:276] 0625 13:52:13.602122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.603066 (+ 944us) service_pool.cc:225] Handling call
0625 13:52:16.906777 (+3303711us) ts_tablet_manager.cc:1145] Deleting tablet 9514bbbda0ba4329a18b0a42e635c6e5
0625 13:52:16.906787 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:16.926177 (+ 19390us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.929300 (+ 3123us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:16.970266 (+ 40966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3303522}
I20250625 13:52:16.983718 1168 tablet_replica.cc:331] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:16.983968 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 9514bbbda0ba4329a18b0a42e635c6e5 (table default.Table_3462a32d0970423c8538a79dae984471 [id=60ccb9d481c14fd68670e5701ce76b84]) successfully deleted
I20250625 13:52:16.984663 2478 raft_consensus.cc:491] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:16.984808 1168 raft_consensus.cc:2241] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:16.984820 2478 raft_consensus.cc:513] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.984969 1168 raft_consensus.cc:2270] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:16.985388 2478 leader_election.cc:290] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:16.985456 1168 ts_tablet_manager.cc:1905] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:16.986130 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c75deb64c74f0282efeed827659ec2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:16.986284 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c75deb64c74f0282efeed827659ec2" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:16.986753 711 leader_election.cc:343] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:16.986968 711 leader_election.cc:343] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:16.987046 711 leader_election.cc:304] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:16.987391 2478 raft_consensus.cc:2747] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:16.987401 1045 raft_consensus.cc:3058] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:16.992060 849 consensus_peers.cc:487] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39ac79102eab46688e767faed6d6f114. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:16.994555 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_231ba9d7a8614e20b45715e4a09b9437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:16.995229 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_231ba9d7a8614e20b45715e4a09b9437 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:16.998082 1171 tablet_bootstrap.cc:492] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:16.998262 1171 ts_tablet_manager.cc:1397] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.003s
I20250625 13:52:16.999030 1171 raft_consensus.cc:357] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:16.999248 1171 raft_consensus.cc:383] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:16.999352 1171 raft_consensus.cc:738] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:16.999614 1171 consensus_queue.cc:260] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.002261 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap replayed 1/1 log segments. Stats: ops{read=699 overwritten=0 applied=675 ignored=0} inserts{seen=643 ignored=0} mutations{seen=762 ignored=0} orphaned_commits=0. Pending: 24 replicates
I20250625 13:52:17.003146 1171 ts_tablet_manager.cc:1428] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 13:52:17.012259 1166 tablet_bootstrap.cc:492] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.013279 1166 ts_tablet_manager.cc:1397] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.144s user 0.017s sys 0.017s
W20250625 13:52:17.014251 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8063f1e6d8a0469f8331ae4de006bf77 consensus state has no leader
I20250625 13:52:17.014295 1166 raft_consensus.cc:357] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:17.015507 1166 raft_consensus.cc:383] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.015714 1166 raft_consensus.cc:738] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.016173 1166 consensus_queue.cc:260] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:17.016347 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 97851.3us
I20250625 13:52:17.016774 2472 raft_consensus.cc:2747] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.017277 1166 ts_tablet_manager.cc:1428] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:17.017994 1166 tablet_bootstrap.cc:492] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.019109 2254 pending_rounds.cc:70] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:17.019333 2254 pending_rounds.cc:77] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520658332991488 op_type: NO_OP noop_request { }
I20250625 13:52:17.019560 2254 raft_consensus.cc:2887] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:17.019731 2254 raft_consensus.cc:2270] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.019949 1166 tablet_bootstrap.cc:654] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.020388 2254 ts_tablet_manager.cc:1905] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.023267 591 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ef2b08fba37f4a38880c018ec2d869c8 (table default.Table_3a4e5f10c2ec453a82fb97800a0d37f7 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]). Expected version 2 got 0
I20250625 13:52:17.024430 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: Bootstrap complete.
I20250625 13:52:17.026492 748 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c75deb64c74f0282efeed827659ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.026723 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a57268a18f4f00a39f8360f9c90737 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.026875 757 tablet_service.cc:1515] Processing DeleteTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.027001 749 tablet_service.cc:1515] Processing DeleteTablet for tablet d9462da7574a46839763b1c388102e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.027285 1165 tablet_replica.cc:331] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:17.027495 1165 raft_consensus.cc:2241] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.027652 1165 raft_consensus.cc:2270] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.028208 1048 raft_consensus.cc:3058] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.028231 1165 ts_tablet_manager.cc:1905] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.031790 2338 raft_consensus.cc:491] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.031934 2338 raft_consensus.cc:513] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.032493 2338 leader_election.cc:290] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.033689 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.034235 849 leader_election.cc:343] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.034723 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:17.034894 775 raft_consensus.cc:2466] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:17.036073 596 catalog_manager.cc:5582] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:17.036473 596 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
I20250625 13:52:17.036794 596 catalog_manager.cc:5582] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:17.036947 851 leader_election.cc:304] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:17.037295 2204 raft_consensus.cc:491] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.037452 2204 raft_consensus.cc:513] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.037977 1165 ts_tablet_manager.cc:1918] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.037997 2204 leader_election.cc:290] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.038116 2204 raft_consensus.cc:2802] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:17.038172 2204 raft_consensus.cc:491] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:17.038233 2204 raft_consensus.cc:3058] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.038307 1165 log.cc:1199] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/f3c75deb64c74f0282efeed827659ec2
I20250625 13:52:17.038633 1165 ts_tablet_manager.cc:1939] T f3c75deb64c74f0282efeed827659ec2 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:17.038710 1168 ts_tablet_manager.cc:1918] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.039674 1165 tablet_replica.cc:331] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:17.039896 1165 raft_consensus.cc:2241] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.040071 1165 raft_consensus.cc:2270] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.040571 1165 ts_tablet_manager.cc:1905] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.041740 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a2007e51d424053bc2836dedf46add2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.042712 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet f3c75deb64c74f0282efeed827659ec2 (table default.Table_40e114808bdd4477ae9ba7f8f92f6f8c [id=b4c1ac962c044cc79f9a8207fee04046]) successfully deleted
W20250625 13:52:17.042949 583 catalog_manager.cc:4895] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 32a57268a18f4f00a39f8360f9c90737 because the tablet was not found. No further retry: Not found: Tablet not found: 32a57268a18f4f00a39f8360f9c90737
I20250625 13:52:17.040537 1171 tablet_bootstrap.cc:492] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.045622 1171 tablet_bootstrap.cc:654] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.040846 2204 raft_consensus.cc:513] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.046854 2204 leader_election.cc:290] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:17.042200 849 leader_election.cc:343] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:17.047253 851 leader_election.cc:343] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9a2007e51d424053bc2836dedf46add2
I20250625 13:52:17.047546 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a029ff3e9b1448d2a8187d7e9cc8c170" } new_table_name: "default.Table_37db3d8c2a0d44e19c1598de8b337d6a" modify_external_catalogs: true
I20250625 13:52:17.047645 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:17.047657 851 leader_election.cc:304] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.048183 2338 raft_consensus.cc:2747] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:17.048499 849 leader_election.cc:343] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.049093 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:17.049268 775 raft_consensus.cc:3058] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.049249 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d942502e3483477a927a158d0a4a3f4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.049911 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d942502e3483477a927a158d0a4a3f4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.051007 1171 tablet_bootstrap.cc:492] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:17.051252 1171 ts_tablet_manager.cc:1397] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250625 13:52:17.051901 775 raft_consensus.cc:2466] T 647c97a8ae484c4a8694e777bc62d602 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:17.052386 851 leader_election.cc:304] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:17.052834 2204 raft_consensus.cc:2802] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:17.052749 1171 raft_consensus.cc:357] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:17.053151 1171 raft_consensus.cc:383] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.053342 1171 raft_consensus.cc:738] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.052992 2204 raft_consensus.cc:695] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:17.053766 2204 consensus_queue.cc:237] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.053887 2338 raft_consensus.cc:491] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.054059 2338 raft_consensus.cc:513] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.054589 2338 leader_election.cc:290] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.054867 1171 consensus_queue.cc:260] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:17.055490 2254 ts_tablet_manager.cc:1918] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:17.055614 2254 log.cc:1199] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/490c8902efeb46c99543747a8787c1c7
I20250625 13:52:17.055974 2254 ts_tablet_manager.cc:1939] T 490c8902efeb46c99543747a8787c1c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:17.056286 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.056947 849 leader_election.cc:343] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.057869 1171 ts_tablet_manager.cc:1428] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:17.061018 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 669de0b7c93a4efea8623dab2e462c8d (table default.Table_8315e81e0d0843819735795f79fa91fb [id=d3092ac7d43a4e9891fe289b304e937c]). Expected version 3 got 0
I20250625 13:52:17.061244 1168 log.cc:1199] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a29708fb688a4a65b84ae2564a040b4c
I20250625 13:52:17.061259 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet e2e6fbf3d81547f798b246a8b8f61387 (table default.Table_14ec8e2e12ed48ebad0ef4224a49e058 [id=7172d4663e37479e9998f617d88f8b6a]). Expected version 2 got 0
I20250625 13:52:17.061357 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (table default.Table_e7d5b0a0e4d64a318b47d1476ac90f2b [id=6903813257eb4459a5b89035cae9e85a]). Expected version 1 got 0
I20250625 13:52:17.061702 1168 ts_tablet_manager.cc:1939] T a29708fb688a4a65b84ae2564a040b4c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:17.062551 2478 raft_consensus.cc:491] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.062709 2478 raft_consensus.cc:513] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:17.063360 2478 leader_election.cc:290] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:17.063473 749 tablet_service.cc:1515] Processing DeleteTablet for tablet afa1c198dd6541809843244c7cc18194 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.063939 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829221b8a79f460d90dd24be6a2d4c43" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:17.064119 1171 tablet_bootstrap.cc:492] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.064109 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:17.064455 711 leader_election.cc:343] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.064678 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ef2c3757c0284aaca70ec317c05195d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.065568 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.065825 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2d6f05348e43339e7167a9cddca2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.066311 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.066586 1171 tablet_bootstrap.cc:654] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.067267 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_11a3ede4b80d4a44bf251533bede8b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.067849 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11a3ede4b80d4a44bf251533bede8b96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.070310 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 34638ea8127a46da9539e8543ea17a91 (DEFAULT_TABLE table=default.Table_1238ef9eebed4e848a72b45103c90092 [id=7d9bc301e1f942c3b7c30cc32364d873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.071198 749 tablet_service.cc:1468] Processing CreateTablet for tablet 34638ea8127a46da9539e8543ea17a91 (DEFAULT_TABLE table=default.Table_1238ef9eebed4e848a72b45103c90092 [id=7d9bc301e1f942c3b7c30cc32364d873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.071506 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:17.071628 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34638ea8127a46da9539e8543ea17a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.071695 775 raft_consensus.cc:2466] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:17.074515 851 leader_election.cc:304] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:17.077566 1171 tablet_bootstrap.cc:492] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:17.078291 2338 raft_consensus.cc:2802] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:17.078217 1171 ts_tablet_manager.cc:1397] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250625 13:52:17.078377 2338 raft_consensus.cc:491] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:17.078438 2338 raft_consensus.cc:3058] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.081024 2338 raft_consensus.cc:513] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.081575 2338 leader_election.cc:290] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.082545 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:17.082757 778 raft_consensus.cc:3058] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.086097 778 raft_consensus.cc:2466] T 38770320fc6541c39e10448978f8a3b0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:17.086591 851 leader_election.cc:304] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:17.086879 2338 raft_consensus.cc:2802] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:17.087013 2338 raft_consensus.cc:695] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:17.087263 2338 consensus_queue.cc:237] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.088332 1171 raft_consensus.cc:357] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.089864 1171 raft_consensus.cc:383] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.090407 1171 raft_consensus.cc:738] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.091360 890 tablet_service.cc:1468] Processing CreateTablet for tablet 34638ea8127a46da9539e8543ea17a91 (DEFAULT_TABLE table=default.Table_1238ef9eebed4e848a72b45103c90092 [id=7d9bc301e1f942c3b7c30cc32364d873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.091872 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34638ea8127a46da9539e8543ea17a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.091827 1171 consensus_queue.cc:260] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.093533 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829221b8a79f460d90dd24be6a2d4c43" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.093641 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1244) took 1650 ms. Trace:
I20250625 13:52:17.093725 2254 rpcz_store.cc:276] 0625 13:52:15.443111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.443279 (+ 168us) service_pool.cc:225] Handling call
0625 13:52:16.972506 (+1529227us) ts_tablet_manager.cc:1145] Deleting tablet 490c8902efeb46c99543747a8787c1c7
0625 13:52:16.972517 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.028487 (+ 55970us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.055370 (+ 26883us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.093616 (+ 38246us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42932,"tablet-delete.queue_time_us":1528907}
I20250625 13:52:17.093883 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:17.094185 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 490c8902efeb46c99543747a8787c1c7 (table default.Table_2a8634c60e08492faffd30a254d0dc11 [id=a0e4a71b13d94a69affc3cc9e0cb5066]) successfully deleted
I20250625 13:52:17.094558 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34638ea8127a46da9539e8543ea17a91. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:17.094702 711 leader_election.cc:343] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.094827 711 leader_election.cc:304] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.095199 2478 raft_consensus.cc:2747] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:17.095518 849 leader_election.cc:343] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.097416 1171 ts_tablet_manager.cc:1428] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20250625 13:52:17.099042 1171 tablet_bootstrap.cc:492] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.103001 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1145) took 3497 ms. Trace:
I20250625 13:52:17.103453 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1260) took 1339 ms. Trace:
I20250625 13:52:17.104302 2254 rpcz_store.cc:276] 0625 13:52:15.764300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.765177 (+ 877us) service_pool.cc:225] Handling call
0625 13:52:17.103330 (+1338153us) ts_tablet_manager.cc:1145] Deleting tablet bc0af0cf4cb54922ad6d1f5de12a46e0
0625 13:52:17.103343 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.103434 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337973}
W20250625 13:52:17.104809 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc0af0cf4cb54922ad6d1f5de12a46e0 already in progress: creating tablet
I20250625 13:52:17.105213 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1259) took 1341 ms. Trace:
I20250625 13:52:17.105384 1168 rpcz_store.cc:276] 0625 13:52:13.606025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.606436 (+ 411us) service_pool.cc:225] Handling call
0625 13:52:16.983659 (+3377223us) ts_tablet_manager.cc:1145] Deleting tablet a29708fb688a4a65b84ae2564a040b4c
0625 13:52:16.983670 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.035541 (+ 51871us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.038633 (+ 3092us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.102984 (+ 64351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3376960}
I20250625 13:52:17.105883 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet a29708fb688a4a65b84ae2564a040b4c (table default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]) successfully deleted
I20250625 13:52:17.107142 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1147) took 3497 ms. Trace:
I20250625 13:52:17.107244 1168 rpcz_store.cc:276] 0625 13:52:13.609379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.623682 (+ 14303us) service_pool.cc:225] Handling call
0625 13:52:17.106985 (+3483303us) ts_tablet_manager.cc:1145] Deleting tablet 20841d3424fb4c48b4de4ed7b2917abe
0625 13:52:17.106995 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.107128 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3483080}
W20250625 13:52:17.107865 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 20841d3424fb4c48b4de4ed7b2917abe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20841d3424fb4c48b4de4ed7b2917abe already in progress: creating tablet
I20250625 13:52:17.108266 1168 tablet_replica.cc:331] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:17.108332 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 54cc2d0cf95b41d6b36483af5436fcde
I20250625 13:52:17.109452 2254 rpcz_store.cc:276] 0625 13:52:15.764214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.767351 (+ 3137us) service_pool.cc:225] Handling call
0625 13:52:17.105109 (+1337758us) ts_tablet_manager.cc:1145] Deleting tablet 2e2711256c9042ebb651ae0ca3d02c89
0625 13:52:17.105117 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.105196 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337573}
W20250625 13:52:17.110383 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2e2711256c9042ebb651ae0ca3d02c89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e2711256c9042ebb651ae0ca3d02c89 already in progress: creating tablet
I20250625 13:52:17.110817 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1266) took 1334 ms. Trace:
I20250625 13:52:17.110329 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 54cc2d0cf95b41d6b36483af5436fcde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.111500 1168 raft_consensus.cc:2241] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:17.111761 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 54cc2d0cf95b41d6b36483af5436fcde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.112004 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 54cc2d0cf95b41d6b36483af5436fcde with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.111791 2254 rpcz_store.cc:276] 0625 13:52:15.776537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.776590 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:17.110701 (+1334111us) ts_tablet_manager.cc:1145] Deleting tablet 9a2007e51d424053bc2836dedf46add2
0625 13:52:17.110709 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.110802 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333966}
I20250625 13:52:17.113050 1171 tablet_bootstrap.cc:654] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.115939 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1c72b998c9314821835754a40d9c88a1" } new_table_name: "default.Table_8d0d72b50bfa43c6979e378ec08f2d12" modify_external_catalogs: true
I20250625 13:52:17.116024 1168 raft_consensus.cc:2270] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.116626 1168 ts_tablet_manager.cc:1905] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:17.118110 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 9a2007e51d424053bc2836dedf46add2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2007e51d424053bc2836dedf46add2 already in progress: creating tablet
I20250625 13:52:17.118594 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1267) took 1341 ms. Trace:
I20250625 13:52:17.119297 2254 rpcz_store.cc:276] 0625 13:52:15.776645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.776884 (+ 239us) service_pool.cc:225] Handling call
0625 13:52:17.118472 (+1341588us) ts_tablet_manager.cc:1145] Deleting tablet d5abd4d83514458c95ec62290b709bc3
0625 13:52:17.118482 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.118575 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341432}
W20250625 13:52:17.120137 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d5abd4d83514458c95ec62290b709bc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5abd4d83514458c95ec62290b709bc3 already in progress: creating tablet
I20250625 13:52:17.120584 1166 tablet_bootstrap.cc:492] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.120774 1166 ts_tablet_manager.cc:1397] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.002s
I20250625 13:52:17.121192 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1258) took 1361 ms. Trace:
I20250625 13:52:17.121520 1166 raft_consensus.cc:357] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.121737 1166 raft_consensus.cc:383] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.121811 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 575043022b1b4ca686775b2aae41a037
I20250625 13:52:17.121824 1166 raft_consensus.cc:738] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.122112 1166 consensus_queue.cc:260] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.122937 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 575043022b1b4ca686775b2aae41a037 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.123165 2254 rpcz_store.cc:276] 0625 13:52:15.760072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.792441 (+ 32369us) service_pool.cc:225] Handling call
0625 13:52:17.121081 (+1328640us) ts_tablet_manager.cc:1145] Deleting tablet 1696cbd156f94d14a5f1da76c8e58584
0625 13:52:17.121090 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.121175 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328372}
W20250625 13:52:17.123548 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 1696cbd156f94d14a5f1da76c8e58584 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1696cbd156f94d14a5f1da76c8e58584 already in progress: creating tablet
I20250625 13:52:17.122942 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 575043022b1b4ca686775b2aae41a037 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.125034 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 575043022b1b4ca686775b2aae41a037 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.125833 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 94dc426fbfeb407a87412d7cfc2d7bb6 (DEFAULT_TABLE table=default.Table_491bf1ae35e74854b2d7f3f716215bbb [id=34c3e009fdfd4b5b88a57d640313b0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.126188 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94dc426fbfeb407a87412d7cfc2d7bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.126225 2254 tablet_replica.cc:331] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:17.126412 2254 raft_consensus.cc:2241] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.126579 2254 raft_consensus.cc:2270] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.126840 1166 ts_tablet_manager.cc:1428] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 13:52:17.127118 1166 tablet_bootstrap.cc:492] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.128747 1166 tablet_bootstrap.cc:654] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.130470 2254 ts_tablet_manager.cc:1905] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.132890 896 tablet_service.cc:1468] Processing CreateTablet for tablet 94dc426fbfeb407a87412d7cfc2d7bb6 (DEFAULT_TABLE table=default.Table_491bf1ae35e74854b2d7f3f716215bbb [id=34c3e009fdfd4b5b88a57d640313b0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.133301 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94dc426fbfeb407a87412d7cfc2d7bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.150306 2162 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Replica starting. Triggering 24 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:17.157227 896 tablet_service.cc:1468] Processing CreateTablet for tablet f8a3f3f7e8164459b718bacd39c66abd (DEFAULT_TABLE table=default.Table_4408ed6957e948b08dee253e7781b84e [id=8fe95b7456994cdcb05c8dd432b919f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.157492 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 8fe150011465428caaa12b99d5ad8dae (DEFAULT_TABLE table=default.Table_01b8d89dbb714bdc83d42185cc4a1fd3 [id=2f413e1c59c147868eecda4e62a2232a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.157666 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a3f3f7e8164459b718bacd39c66abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.157882 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe150011465428caaa12b99d5ad8dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.164853 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_13065c9f25c443f4b762794456e59b3e" } new_table_name: "default.Table_b0e2772ddddc4faf81571778a3405e88" modify_external_catalogs: true
I20250625 13:52:17.141957 749 tablet_service.cc:1468] Processing CreateTablet for tablet 94dc426fbfeb407a87412d7cfc2d7bb6 (DEFAULT_TABLE table=default.Table_491bf1ae35e74854b2d7f3f716215bbb [id=34c3e009fdfd4b5b88a57d640313b0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.155933 757 tablet_service.cc:1468] Processing CreateTablet for tablet f8a3f3f7e8164459b718bacd39c66abd (DEFAULT_TABLE table=default.Table_4408ed6957e948b08dee253e7781b84e [id=8fe95b7456994cdcb05c8dd432b919f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.167995 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a3f3f7e8164459b718bacd39c66abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.168253 1171 tablet_bootstrap.cc:492] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:17.169099 1171 ts_tablet_manager.cc:1397] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.001s
I20250625 13:52:17.171455 1168 ts_tablet_manager.cc:1918] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:17.171854 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94dc426fbfeb407a87412d7cfc2d7bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.177055 2254 ts_tablet_manager.cc:1918] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.174995 1171 raft_consensus.cc:357] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.177328 1168 log.cc:1199] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/6d95a376304e464ab75e2e1e00e7a924
I20250625 13:52:17.178066 1168 ts_tablet_manager.cc:1939] T 6d95a376304e464ab75e2e1e00e7a924 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:17.179092 2254 log.cc:1199] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/7ba1338d0562418eadabdb8dbcaedee2
I20250625 13:52:17.183651 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1153) took 3540 ms. Trace:
I20250625 13:52:17.184074 2254 ts_tablet_manager.cc:1939] T 7ba1338d0562418eadabdb8dbcaedee2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:17.184470 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1271) took 1354 ms. Trace:
I20250625 13:52:17.184599 1168 rpcz_store.cc:276] 0625 13:52:13.642857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.648439 (+ 5582us) service_pool.cc:225] Handling call
0625 13:52:17.108209 (+3459770us) ts_tablet_manager.cc:1145] Deleting tablet 6d95a376304e464ab75e2e1e00e7a924
0625 13:52:17.108222 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.136724 (+ 28502us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.171333 (+ 34609us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.183634 (+ 12301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3458699}
I20250625 13:52:17.185109 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 6d95a376304e464ab75e2e1e00e7a924 (table default.Table_b2017c57e5c7416494c31f6dcefdc43d [id=8c674ba68eb74eb8b6646ff2ef9c1561]) successfully deleted
I20250625 13:52:17.177384 1171 raft_consensus.cc:383] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.185364 1171 raft_consensus.cc:738] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.185652 1171 consensus_queue.cc:260] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.186206 1171 ts_tablet_manager.cc:1428] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 13:52:17.186523 1171 tablet_bootstrap.cc:492] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.188060 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6352e917532840ca832e32ddf0747da4
I20250625 13:52:17.186906 2254 rpcz_store.cc:276] 0625 13:52:15.830087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.830225 (+ 138us) service_pool.cc:225] Handling call
0625 13:52:17.126182 (+1295957us) ts_tablet_manager.cc:1145] Deleting tablet 7ba1338d0562418eadabdb8dbcaedee2
0625 13:52:17.126191 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.152070 (+ 25879us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.176916 (+ 24846us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.184452 (+ 7536us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295770}
I20250625 13:52:17.186816 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1152) took 3550 ms. Trace:
I20250625 13:52:17.189154 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 6352e917532840ca832e32ddf0747da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.189599 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 7ba1338d0562418eadabdb8dbcaedee2 (table default.Table_c7891a214410484c907ee024a3f40331 [id=21a2a642fff348518e79d4e4db1f32ad]) successfully deleted
I20250625 13:52:17.188319 1168 rpcz_store.cc:276] 0625 13:52:13.635934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.649934 (+ 14000us) service_pool.cc:225] Handling call
0625 13:52:17.186676 (+3536742us) ts_tablet_manager.cc:1145] Deleting tablet 9f24ae7c08bd42959ab48f7007ec98d3
0625 13:52:17.186685 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.186801 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3536204}
I20250625 13:52:17.190094 2254 tablet_replica.cc:331] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:17.190402 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9f24ae7c08bd42959ab48f7007ec98d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f24ae7c08bd42959ab48f7007ec98d3 already in progress: creating tablet
I20250625 13:52:17.191267 2254 raft_consensus.cc:2241] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.192029 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 6352e917532840ca832e32ddf0747da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.193353 2514 tablet.cc:1722] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:17.193955 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 15c81fab8e0b442e8d39556c1bce742e (table default.Table_b32fea3ffed64728b8b1b4a86b44c628 [id=4d9dc7e6615940e38f7c61644a03022a]). Expected version 1 got 0
I20250625 13:52:17.194674 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_250cc469d51d495287ef1ff5cd5467e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.195192 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_250cc469d51d495287ef1ff5cd5467e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.195485 2271 raft_consensus.cc:491] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.195811 2271 raft_consensus.cc:513] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.196509 2271 leader_election.cc:290] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:17.198948 851 leader_election.cc:343] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8e2bc4df39674dc3be0cd32c68f1ebca
I20250625 13:52:17.201716 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a6b2f861d63d4299b3e472cf705a1b67
I20250625 13:52:17.203017 1166 tablet_bootstrap.cc:492] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.203147 1166 ts_tablet_manager.cc:1397] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20250625 13:52:17.203984 1166 raft_consensus.cc:357] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.206141 757 tablet_service.cc:1515] Processing DeleteTablet for tablet a6b2f861d63d4299b3e472cf705a1b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.207386 750 tablet_service.cc:1468] Processing CreateTablet for tablet 8fe150011465428caaa12b99d5ad8dae (DEFAULT_TABLE table=default.Table_01b8d89dbb714bdc83d42185cc4a1fd3 [id=2f413e1c59c147868eecda4e62a2232a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.207671 1166 raft_consensus.cc:383] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.207751 1166 raft_consensus.cc:738] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.207731 896 tablet_service.cc:1515] Processing DeleteTablet for tablet a6b2f861d63d4299b3e472cf705a1b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.207839 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe150011465428caaa12b99d5ad8dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.208523 1166 consensus_queue.cc:260] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.211763 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1159) took 3384 ms. Trace:
I20250625 13:52:17.212853 2254 raft_consensus.cc:2270] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.212983 1168 rpcz_store.cc:276] 0625 13:52:13.827176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.827352 (+ 176us) service_pool.cc:225] Handling call
0625 13:52:17.190646 (+3363294us) ts_tablet_manager.cc:1145] Deleting tablet 6c331c57bff44091be2849c21d6ab2d7
0625 13:52:17.190655 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.211718 (+ 21063us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3363139}
I20250625 13:52:17.213851 2254 ts_tablet_manager.cc:1905] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:17.215487 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6c331c57bff44091be2849c21d6ab2d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c331c57bff44091be2849c21d6ab2d7 already in progress: creating tablet
I20250625 13:52:17.215958 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1158) took 3390 ms. Trace:
I20250625 13:52:17.219362 1168 rpcz_store.cc:276] 0625 13:52:13.825154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.834172 (+ 9018us) service_pool.cc:225] Handling call
0625 13:52:17.215818 (+3381646us) ts_tablet_manager.cc:1145] Deleting tablet 55a254d82f764aa2860e465b4857a413
0625 13:52:17.215831 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.215939 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3381431}
I20250625 13:52:17.197624 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e2bc4df39674dc3be0cd32c68f1ebca" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.203656 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet a6b2f861d63d4299b3e472cf705a1b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.191821 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6352e917532840ca832e32ddf0747da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.220108 2506 tablet.cc:1722] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250625 13:52:17.220153 849 leader_election.cc:343] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.220343 849 leader_election.cc:304] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.220625 2338 raft_consensus.cc:2747] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.220993 1166 ts_tablet_manager.cc:1428] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
W20250625 13:52:17.221185 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 55a254d82f764aa2860e465b4857a413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55a254d82f764aa2860e465b4857a413 already in progress: creating tablet
I20250625 13:52:17.221613 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1160) took 3377 ms. Trace:
I20250625 13:52:17.221886 1168 rpcz_store.cc:276] 0625 13:52:13.844295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.844339 (+ 44us) service_pool.cc:225] Handling call
0625 13:52:17.221494 (+3377155us) ts_tablet_manager.cc:1145] Deleting tablet 6fcc17f839504d6b85526b638763ec5b
0625 13:52:17.221507 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.221594 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3377008}
I20250625 13:52:17.222303 2515 tablet.cc:1722] T 86af0ea6648045a1ac4ba903d20849b7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:17.226846 750 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5dc19bae22418792e1563a0ed21d1e (DEFAULT_TABLE table=default.Table_190dea7daa454753b901b65cd213f592 [id=4c0aa0b19c18488f8a3ba2b2dc9a688a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.227248 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5dc19bae22418792e1563a0ed21d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.231950 757 tablet_service.cc:1468] Processing CreateTablet for tablet 0c88fcbea4654b9c9dcd27527e3841d4 (DEFAULT_TABLE table=default.Table_df9f76474cdb40fdb91c03605ebdee8f [id=e7305921cab7405893184984a6e435e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.232299 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c88fcbea4654b9c9dcd27527e3841d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.234042 1166 tablet_bootstrap.cc:492] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.234333 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 0c88fcbea4654b9c9dcd27527e3841d4 (DEFAULT_TABLE table=default.Table_df9f76474cdb40fdb91c03605ebdee8f [id=e7305921cab7405893184984a6e435e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.234422 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 078a05d5168a46698535a8d3fd08f634 (DEFAULT_TABLE table=default.Table_eb5680877f724ba893b7661d6684834f [id=c47015b938c247768995f1c0c7b87b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.234773 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c88fcbea4654b9c9dcd27527e3841d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.235932 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078a05d5168a46698535a8d3fd08f634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.238339 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5dc19bae22418792e1563a0ed21d1e (DEFAULT_TABLE table=default.Table_190dea7daa454753b901b65cd213f592 [id=4c0aa0b19c18488f8a3ba2b2dc9a688a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.238685 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5dc19bae22418792e1563a0ed21d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.242048 1166 tablet_bootstrap.cc:654] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.245114 2472 raft_consensus.cc:491] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.245247 2472 raft_consensus.cc:513] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.245862 2472 leader_election.cc:290] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:17.246986 985 leader_election.cc:343] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3859f399a7b84de99658380bd6c24ff5
I20250625 13:52:17.253494 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 15c81fab8e0b442e8d39556c1bce742e (table default.Table_b32fea3ffed64728b8b1b4a86b44c628 [id=4d9dc7e6615940e38f7c61644a03022a]). Expected version 1 got 0
I20250625 13:52:17.254703 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bd28610323b44b249068082bd830bd60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.255362 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd28610323b44b249068082bd830bd60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:17.276419 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6fcc17f839504d6b85526b638763ec5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fcc17f839504d6b85526b638763ec5b already in progress: creating tablet
I20250625 13:52:17.221836 1171 tablet_bootstrap.cc:654] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.235317 891 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5dc19bae22418792e1563a0ed21d1e (DEFAULT_TABLE table=default.Table_190dea7daa454753b901b65cd213f592 [id=4c0aa0b19c18488f8a3ba2b2dc9a688a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.277076 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5dc19bae22418792e1563a0ed21d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.278407 1165 ts_tablet_manager.cc:1918] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.278532 1165 log.cc:1199] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/db6e9db2375144b38ec43ffca9ca4c1e
I20250625 13:52:17.278870 1165 ts_tablet_manager.cc:1939] T db6e9db2375144b38ec43ffca9ca4c1e P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:17.235656 882 tablet_service.cc:1468] Processing CreateTablet for tablet 0c88fcbea4654b9c9dcd27527e3841d4 (DEFAULT_TABLE table=default.Table_df9f76474cdb40fdb91c03605ebdee8f [id=e7305921cab7405893184984a6e435e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.246780 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3859f399a7b84de99658380bd6c24ff5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:17.279747 915 raft_consensus.cc:2466] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:17.279711 1030 tablet_service.cc:1468] Processing CreateTablet for tablet f8a3f3f7e8164459b718bacd39c66abd (DEFAULT_TABLE table=default.Table_4408ed6957e948b08dee253e7781b84e [id=8fe95b7456994cdcb05c8dd432b919f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.279891 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c88fcbea4654b9c9dcd27527e3841d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.253021 896 tablet_service.cc:1468] Processing CreateTablet for tablet 8fe150011465428caaa12b99d5ad8dae (DEFAULT_TABLE table=default.Table_01b8d89dbb714bdc83d42185cc4a1fd3 [id=2f413e1c59c147868eecda4e62a2232a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.268584 2271 raft_consensus.cc:491] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.280085 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a3f3f7e8164459b718bacd39c66abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.280031 2271 raft_consensus.cc:513] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.281025 2271 leader_election.cc:290] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.281322 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe150011465428caaa12b99d5ad8dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.283710 983 leader_election.cc:304] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.284093 1170 tablet_bootstrap.cc:492] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:17.284101 2472 raft_consensus.cc:2802] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:17.284433 2472 raft_consensus.cc:491] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:17.284498 2472 raft_consensus.cc:3058] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.287091 2472 raft_consensus.cc:513] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:17.287864 851 leader_election.cc:343] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 32a57268a18f4f00a39f8360f9c90737
I20250625 13:52:17.289870 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a57268a18f4f00a39f8360f9c90737" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.291213 849 leader_election.cc:343] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.291383 849 leader_election.cc:304] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.291846 2271 raft_consensus.cc:2747] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:17.292876 985 leader_election.cc:343] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3859f399a7b84de99658380bd6c24ff5
I20250625 13:52:17.292969 594 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet df72a1a461aa44fe80cff6e3870da8e7 (table default.Table_635e90a4272746cab8e8f9802215e784 [id=1ca6396d145c4b68a68e153cf78f8bcc]). Expected version 1 got 0
I20250625 13:52:17.293067 2472 leader_election.cc:290] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.293612 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3859f399a7b84de99658380bd6c24ff5" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:17.293792 915 raft_consensus.cc:3058] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.296314 915 raft_consensus.cc:2466] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:17.296754 983 leader_election.cc:304] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.297412 2513 tablet.cc:1722] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:17.297925 2472 raft_consensus.cc:2802] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:17.298193 2472 raft_consensus.cc:695] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:17.298472 2472 consensus_queue.cc:237] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.298864 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet aef43c4962c747dbb43039d1e706bbd9
I20250625 13:52:17.284356 1170 ts_tablet_manager.cc:1397] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.363s user 0.024s sys 0.059s
I20250625 13:52:17.301923 748 tablet_service.cc:1515] Processing DeleteTablet for tablet aef43c4962c747dbb43039d1e706bbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.301980 1170 raft_consensus.cc:357] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.305379 882 tablet_service.cc:1515] Processing DeleteTablet for tablet aef43c4962c747dbb43039d1e706bbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.302592 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet aef43c4962c747dbb43039d1e706bbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:16 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.305625 1170 raft_consensus.cc:383] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.305732 1170 raft_consensus.cc:738] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.305976 1170 consensus_queue.cc:260] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.306785 1170 ts_tablet_manager.cc:1428] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 13:52:17.307052 1170 tablet_bootstrap.cc:492] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:17.308454 1170 tablet_bootstrap.cc:654] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.312059 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 49b81b7f127d4f2c89ecbecb8a419e5b (DEFAULT_TABLE table=default.Table_13ab9b8203fe4404afbd33bdd16d06e6 [id=16e77db14c534086857f3c226330b65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.312553 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b81b7f127d4f2c89ecbecb8a419e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.315543 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1157) took 3494 ms. Trace:
I20250625 13:52:17.315654 1168 rpcz_store.cc:276] 0625 13:52:13.821503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.852422 (+ 30919us) service_pool.cc:225] Handling call
0625 13:52:17.315423 (+3463001us) ts_tablet_manager.cc:1145] Deleting tablet d35517bbf1cb4695ae46094dc883175f
0625 13:52:17.315431 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.315524 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462053}
W20250625 13:52:17.316321 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d35517bbf1cb4695ae46094dc883175f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d35517bbf1cb4695ae46094dc883175f already in progress: creating tablet
I20250625 13:52:17.318660 882 tablet_service.cc:1468] Processing CreateTablet for tablet 078a05d5168a46698535a8d3fd08f634 (DEFAULT_TABLE table=default.Table_eb5680877f724ba893b7661d6684834f [id=c47015b938c247768995f1c0c7b87b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.319006 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078a05d5168a46698535a8d3fd08f634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.322214 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1161) took 3463 ms. Trace:
I20250625 13:52:17.322335 1168 rpcz_store.cc:276] 0625 13:52:13.858952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.861204 (+ 2252us) service_pool.cc:225] Handling call
0625 13:52:17.321895 (+3460691us) ts_tablet_manager.cc:1145] Deleting tablet a7b72345e04445f7a393c469a348d126
0625 13:52:17.322110 (+ 215us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.322196 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3459375}
W20250625 13:52:17.322804 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet a7b72345e04445f7a393c469a348d126 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7b72345e04445f7a393c469a348d126 already in progress: creating tablet
I20250625 13:52:17.324026 1171 tablet_bootstrap.cc:492] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:17.324188 1171 ts_tablet_manager.cc:1397] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.000s
I20250625 13:52:17.324853 1170 tablet_bootstrap.cc:492] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:17.325069 593 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 (table default.Table_d3e067c08fee43e69098a905cf6418aa [id=bd5767cb7ca1457a9f41dbad6535b38b]). Expected version 3 got 0
I20250625 13:52:17.325268 1170 ts_tablet_manager.cc:1397] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20250625 13:52:17.325799 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 2593db823e974f1eb653470d5a23338f from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:17.326279 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d069adc5ec614e949d7f2286f54e97a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:17.325078 1168 tablet_replica.cc:331] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:17.326601 1168 raft_consensus.cc:2241] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.326761 1168 raft_consensus.cc:2270] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:17.326751 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d069adc5ec614e949d7f2286f54e97a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.327268 1168 ts_tablet_manager.cc:1905] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.327579 2516 tablet.cc:1722] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 13:52:17.333778 2478 raft_consensus.cc:491] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.333937 2478 raft_consensus.cc:513] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.334564 2478 leader_election.cc:290] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.335455 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2d6f05348e43339e7167a9cddca2ba" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:17.336061 711 leader_election.cc:343] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.341090 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2d6f05348e43339e7167a9cddca2ba" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.341498 748 tablet_service.cc:1468] Processing CreateTablet for tablet 49b81b7f127d4f2c89ecbecb8a419e5b (DEFAULT_TABLE table=default.Table_13ab9b8203fe4404afbd33bdd16d06e6 [id=16e77db14c534086857f3c226330b65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.341895 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b81b7f127d4f2c89ecbecb8a419e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.342566 891 tablet_service.cc:1515] Processing DeleteTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 with delete_type TABLET_DATA_DELETED (replaced by tablet 879d91dcfb0b4868a7ba6833c211e5eb) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.342774 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.343120 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca with delete_type TABLET_DATA_DELETED (replaced by tablet 67e05cc542384b4b8117a1d9330db24b) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:17.341569 711 leader_election.cc:343] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.343370 711 leader_election.cc:304] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.343636 1170 raft_consensus.cc:357] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:17.343780 2478 raft_consensus.cc:2747] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.344111 1171 raft_consensus.cc:357] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.343858 2162 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fec9d8a79054a5991ae90d8a3101e32, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.348122 2162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 675, Last appended: 1.699, Last appended by leader: 699, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:17.349941 1171 raft_consensus.cc:383] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.350080 1170 raft_consensus.cc:383] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.351794 748 tablet_service.cc:1468] Processing CreateTablet for tablet 53e9c9b3dbbb4b5bb29bfe78a0dbe443 (DEFAULT_TABLE table=default.Table_930049e3536b44a1975c5f98bff68581 [id=ce9c6015e6df401d89eb165ba91acee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.351974 1171 raft_consensus.cc:738] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.352092 1170 raft_consensus.cc:738] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.352156 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e9c9b3dbbb4b5bb29bfe78a0dbe443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.352376 1170 consensus_queue.cc:260] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:17.351770 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 53e9c9b3dbbb4b5bb29bfe78a0dbe443 (DEFAULT_TABLE table=default.Table_930049e3536b44a1975c5f98bff68581 [id=ce9c6015e6df401d89eb165ba91acee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.356323 896 tablet_service.cc:1468] Processing CreateTablet for tablet 49b81b7f127d4f2c89ecbecb8a419e5b (DEFAULT_TABLE table=default.Table_13ab9b8203fe4404afbd33bdd16d06e6 [id=16e77db14c534086857f3c226330b65c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.358229 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e9c9b3dbbb4b5bb29bfe78a0dbe443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.358361 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49b81b7f127d4f2c89ecbecb8a419e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.363086 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 68b8c8aa8036454298bf1e81bab91c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.363420 1171 consensus_queue.cc:260] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.363976 1171 ts_tablet_manager.cc:1428] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20250625 13:52:17.364296 1171 tablet_bootstrap.cc:492] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.364121 596 catalog_manager.cc:5582] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:17.364782 596 catalog_manager.cc:5582] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:17.365120 596 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 38770320fc6541c39e10448978f8a3b0 (table default.Table_6d9870087cdc4dfca8fa438020887d80 [id=43487ee990e640bcbeefc7c142075998]). Expected version 2 got 0
I20250625 13:52:17.365428 1170 ts_tablet_manager.cc:1428] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250625 13:52:17.368737 2472 raft_consensus.cc:491] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.368865 2472 raft_consensus.cc:513] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.369488 2472 leader_election.cc:290] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.370380 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:52:17.370929 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b610cb974e8c45eeb756278c193c5357 consensus state has no leader
I20250625 13:52:17.372395 1170 tablet_bootstrap.cc:492] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:17.374298 1170 tablet_bootstrap.cc:654] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.376538 2478 raft_consensus.cc:491] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.376691 2478 raft_consensus.cc:513] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.377285 2478 leader_election.cc:290] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.371241 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b54183cf36b049c7bed30deb1604c59a" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:17.378204 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2e6fbf3d81547f798b246a8b8f61387" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:17.378700 711 leader_election.cc:343] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.380694 1171 tablet_bootstrap.cc:654] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.377957 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2e6fbf3d81547f798b246a8b8f61387" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.378576 2472 raft_consensus.cc:491] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.386623 2472 raft_consensus.cc:513] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.386874 749 tablet_service.cc:1468] Processing CreateTablet for tablet 078a05d5168a46698535a8d3fd08f634 (DEFAULT_TABLE table=default.Table_eb5680877f724ba893b7661d6684834f [id=c47015b938c247768995f1c0c7b87b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.387301 2472 leader_election.cc:290] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.378640 983 leader_election.cc:304] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:17.388571 985 leader_election.cc:343] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 723246e1e6824faf9f9d39c3ea941fe4
I20250625 13:52:17.388690 1168 ts_tablet_manager.cc:1918] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.388768 2472 raft_consensus.cc:2747] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.388908 1168 log.cc:1199] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/68b8c8aa8036454298bf1e81bab91c01
I20250625 13:52:17.389226 1168 ts_tablet_manager.cc:1939] T 68b8c8aa8036454298bf1e81bab91c01 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:17.389423 711 leader_election.cc:343] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.389533 711 leader_election.cc:304] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:17.390014 983 leader_election.cc:343] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 723246e1e6824faf9f9d39c3ea941fe4
I20250625 13:52:17.390115 983 leader_election.cc:304] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.390220 2478 raft_consensus.cc:2747] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.390409 2472 raft_consensus.cc:2747] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.379428 2254 ts_tablet_manager.cc:1918] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.390683 2254 log.cc:1199] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/74884bf313724d678d3b5f5869f09f97
I20250625 13:52:17.391028 2254 ts_tablet_manager.cc:1939] T 74884bf313724d678d3b5f5869f09f97 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:17.391343 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1273) took 1338 ms. Trace:
I20250625 13:52:17.391459 2254 rpcz_store.cc:276] 0625 13:52:16.052799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.052860 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:17.190040 (+1137180us) ts_tablet_manager.cc:1145] Deleting tablet 74884bf313724d678d3b5f5869f09f97
0625 13:52:17.190052 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.344958 (+154906us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.379335 (+ 34377us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.391329 (+ 11994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136957}
I20250625 13:52:17.391875 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1274) took 1151 ms. Trace:
I20250625 13:52:17.391970 2254 rpcz_store.cc:276] 0625 13:52:16.240524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.240600 (+ 76us) service_pool.cc:225] Handling call
0625 13:52:17.391777 (+1151177us) ts_tablet_manager.cc:1145] Deleting tablet 5d467dbf133147e7a517c6fc4dff433a
0625 13:52:17.391785 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.391862 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151013}
I20250625 13:52:17.392235 2254 tablet_replica.cc:331] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:17.392498 2254 raft_consensus.cc:2241] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.392681 2254 raft_consensus.cc:2270] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.393194 2254 ts_tablet_manager.cc:1905] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.394129 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 74884bf313724d678d3b5f5869f09f97 (table default.Table_25d9c73fdf1b43b787345769f54915d6 [id=79ace8a081c447a5b4c44b5911a906c8]) successfully deleted
W20250625 13:52:17.394469 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 5d467dbf133147e7a517c6fc4dff433a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d467dbf133147e7a517c6fc4dff433a already in progress: creating tablet
I20250625 13:52:17.395252 2472 consensus_queue.cc:1035] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:17.396437 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1162) took 3445 ms. Trace:
I20250625 13:52:17.396958 1168 rpcz_store.cc:276] 0625 13:52:13.951187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:13.952747 (+ 1560us) service_pool.cc:225] Handling call
0625 13:52:17.325016 (+3372269us) ts_tablet_manager.cc:1145] Deleting tablet 68b8c8aa8036454298bf1e81bab91c01
0625 13:52:17.325025 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.330217 (+ 5192us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.388594 (+ 58377us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.396419 (+ 7825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3372074}
I20250625 13:52:17.397850 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 68b8c8aa8036454298bf1e81bab91c01 (table default.Table_068bedd508da462681c0138ddd8964e3 [id=3a4d573b5eb84c639bf289ec308eb50b]) successfully deleted
I20250625 13:52:17.398340 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1164) took 3267 ms. Trace:
I20250625 13:52:17.398783 1168 rpcz_store.cc:276] 0625 13:52:14.130718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.130788 (+ 70us) service_pool.cc:225] Handling call
0625 13:52:17.398199 (+3267411us) ts_tablet_manager.cc:1145] Deleting tablet 19f1050a29b540558aaf1e31118e4531
0625 13:52:17.398211 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.398319 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3267213}
I20250625 13:52:17.398998 2254 ts_tablet_manager.cc:1918] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.399912 1166 tablet_bootstrap.cc:492] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.400074 1166 ts_tablet_manager.cc:1397] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.166s user 0.006s sys 0.009s
W20250625 13:52:17.400156 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 19f1050a29b540558aaf1e31118e4531 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19f1050a29b540558aaf1e31118e4531 already in progress: creating tablet
I20250625 13:52:17.400710 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1165) took 3266 ms. Trace:
I20250625 13:52:17.400916 1166 raft_consensus.cc:357] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.401116 1166 raft_consensus.cc:383] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.401206 1166 raft_consensus.cc:738] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.401441 1166 consensus_queue.cc:260] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.401250 1168 rpcz_store.cc:276] 0625 13:52:14.134274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.134880 (+ 606us) service_pool.cc:225] Handling call
0625 13:52:17.400576 (+3265696us) ts_tablet_manager.cc:1145] Deleting tablet 1f47260bf0ad41fdbfbad50380a12547
0625 13:52:17.400591 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.400688 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3265482}
W20250625 13:52:17.401865 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1f47260bf0ad41fdbfbad50380a12547 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f47260bf0ad41fdbfbad50380a12547 already in progress: creating tablet
I20250625 13:52:17.401988 1166 ts_tablet_manager.cc:1428] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:17.404255 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1166) took 3264 ms. Trace:
I20250625 13:52:17.404619 1168 rpcz_store.cc:276] 0625 13:52:14.139975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.140447 (+ 472us) service_pool.cc:225] Handling call
0625 13:52:17.404145 (+3263698us) ts_tablet_manager.cc:1145] Deleting tablet 2b29392cfeb14f46a60fd0731f220127
0625 13:52:17.404157 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.404239 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3257658}
W20250625 13:52:17.405481 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2b29392cfeb14f46a60fd0731f220127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b29392cfeb14f46a60fd0731f220127 already in progress: creating tablet
I20250625 13:52:17.406390 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1167) took 3260 ms. Trace:
I20250625 13:52:17.406915 1168 rpcz_store.cc:276] 0625 13:52:14.145906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.146114 (+ 208us) service_pool.cc:225] Handling call
0625 13:52:17.406267 (+3260153us) ts_tablet_manager.cc:1145] Deleting tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc
0625 13:52:17.406277 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.406371 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3259634}
W20250625 13:52:17.407539 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3dcc3b1720074bcb9dd18c16b0cd6bcc already in progress: creating tablet
I20250625 13:52:17.407972 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1168) took 3246 ms. Trace:
I20250625 13:52:17.411409 1168 rpcz_store.cc:276] 0625 13:52:14.161158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.166634 (+ 5476us) service_pool.cc:225] Handling call
0625 13:52:17.407871 (+3241237us) ts_tablet_manager.cc:1145] Deleting tablet d7f68a0c4bb341a9a90cc2be0f78daca
0625 13:52:17.407881 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.407955 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3240799}
W20250625 13:52:17.412144 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d7f68a0c4bb341a9a90cc2be0f78daca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7f68a0c4bb341a9a90cc2be0f78daca already in progress: creating tablet
I20250625 13:52:17.413584 758 tablet_service.cc:1468] Processing CreateTablet for tablet aafb7eef813f48fc8295fbb30a29cd0d (DEFAULT_TABLE table=default.Table_129a669a3cfe4cffbe85f526c5d047d5 [id=e3ee2885822c46f7a3347a7d3f99ebb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.414196 1166 tablet_bootstrap.cc:492] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.414938 890 tablet_service.cc:1468] Processing CreateTablet for tablet 53e9c9b3dbbb4b5bb29bfe78a0dbe443 (DEFAULT_TABLE table=default.Table_930049e3536b44a1975c5f98bff68581 [id=ce9c6015e6df401d89eb165ba91acee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.415359 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e9c9b3dbbb4b5bb29bfe78a0dbe443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.415675 746 tablet_service.cc:1468] Processing CreateTablet for tablet 946c48642da947e08af0448a6bb79db8 (DEFAULT_TABLE table=default.Table_12d22cb86cd5435c84a242d6e0eb12de [id=c81f7a05f1de4e6abbafbf00f95e398b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.416061 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8a9b6e6e1f2d41288a958115dc18d4fb (DEFAULT_TABLE table=default.Table_c52f47756979445e9c05f1f68d2283cd [id=b1eb581e5cc74659a2bdfbd8d3aa9cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.416152 745 tablet_service.cc:1468] Processing CreateTablet for tablet dd6979b2479147ba94842159f0b79c73 (DEFAULT_TABLE table=default.Table_8f3b560e7437430287394d43472d24fb [id=9e7344adacda44cca5c64ea0a1560fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.417268 742 tablet_service.cc:1468] Processing CreateTablet for tablet 86588eb3c34c4536b14f76132bdf3adf (DEFAULT_TABLE table=default.Table_685641e78e7d43b0a6432cd7f23aa34e [id=97aa8e3bb41e4ff680f1715177252154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.418819 753 tablet_service.cc:1468] Processing CreateTablet for tablet d406206ea8e14121bf43214adfc5afda (DEFAULT_TABLE table=default.Table_98ed87f3ecee4add88ff785fe7175ce1 [id=fb0343dcc81d4068b7faab7989c391fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.423892 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078a05d5168a46698535a8d3fd08f634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.424166 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafb7eef813f48fc8295fbb30a29cd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.424407 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d406206ea8e14121bf43214adfc5afda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.424564 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6979b2479147ba94842159f0b79c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.425017 891 tablet_service.cc:1468] Processing CreateTablet for tablet 946c48642da947e08af0448a6bb79db8 (DEFAULT_TABLE table=default.Table_12d22cb86cd5435c84a242d6e0eb12de [id=c81f7a05f1de4e6abbafbf00f95e398b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.425256 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946c48642da947e08af0448a6bb79db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.425343 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946c48642da947e08af0448a6bb79db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.425812 894 tablet_service.cc:1468] Processing CreateTablet for tablet dd6979b2479147ba94842159f0b79c73 (DEFAULT_TABLE table=default.Table_8f3b560e7437430287394d43472d24fb [id=9e7344adacda44cca5c64ea0a1560fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.426101 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6979b2479147ba94842159f0b79c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.426632 2478 raft_consensus.cc:491] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.426756 2478 raft_consensus.cc:513] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.427181 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9b6e6e1f2d41288a958115dc18d4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.427536 886 tablet_service.cc:1468] Processing CreateTablet for tablet 8a9b6e6e1f2d41288a958115dc18d4fb (DEFAULT_TABLE table=default.Table_c52f47756979445e9c05f1f68d2283cd [id=b1eb581e5cc74659a2bdfbd8d3aa9cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.427831 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9b6e6e1f2d41288a958115dc18d4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.406967 2523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:17.429323 2523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:52:17.431128 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86588eb3c34c4536b14f76132bdf3adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.431617 889 tablet_service.cc:1468] Processing CreateTablet for tablet 86588eb3c34c4536b14f76132bdf3adf (DEFAULT_TABLE table=default.Table_685641e78e7d43b0a6432cd7f23aa34e [id=97aa8e3bb41e4ff680f1715177252154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.431953 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86588eb3c34c4536b14f76132bdf3adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.436637 1166 tablet_bootstrap.cc:654] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.437616 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet db6e9db2375144b38ec43ffca9ca4c1e (table default.Table_20f874f96e2d4a3089c21ff64313c62a [id=d1c04c15d77547f4a295ad46828c754e]) successfully deleted
I20250625 13:52:17.438069 1165 tablet_replica.cc:331] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:17.438300 1165 raft_consensus.cc:2241] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.438532 1165 raft_consensus.cc:2270] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.439118 1165 ts_tablet_manager.cc:1905] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.439499 2162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:52:17.447553 2478 leader_election.cc:290] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.448253 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:17.448822 711 leader_election.cc:343] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.422586 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 946c48642da947e08af0448a6bb79db8 (DEFAULT_TABLE table=default.Table_12d22cb86cd5435c84a242d6e0eb12de [id=c81f7a05f1de4e6abbafbf00f95e398b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.422763 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d406206ea8e14121bf43214adfc5afda (DEFAULT_TABLE table=default.Table_98ed87f3ecee4add88ff785fe7175ce1 [id=fb0343dcc81d4068b7faab7989c391fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.451074 2254 log.cc:1199] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/9514bbbda0ba4329a18b0a42e635c6e5
I20250625 13:52:17.451367 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946c48642da947e08af0448a6bb79db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.423532 1024 tablet_service.cc:1468] Processing CreateTablet for tablet dd6979b2479147ba94842159f0b79c73 (DEFAULT_TABLE table=default.Table_8f3b560e7437430287394d43472d24fb [id=9e7344adacda44cca5c64ea0a1560fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.423831 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 86588eb3c34c4536b14f76132bdf3adf (DEFAULT_TABLE table=default.Table_685641e78e7d43b0a6432cd7f23aa34e [id=97aa8e3bb41e4ff680f1715177252154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.422763 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 8a9b6e6e1f2d41288a958115dc18d4fb (DEFAULT_TABLE table=default.Table_c52f47756979445e9c05f1f68d2283cd [id=b1eb581e5cc74659a2bdfbd8d3aa9cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.448299 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.452273 711 leader_election.cc:343] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.452389 711 leader_election.cc:304] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.452729 896 tablet_service.cc:1468] Processing CreateTablet for tablet aafb7eef813f48fc8295fbb30a29cd0d (DEFAULT_TABLE table=default.Table_129a669a3cfe4cffbe85f526c5d047d5 [id=e3ee2885822c46f7a3347a7d3f99ebb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.452813 2478 raft_consensus.cc:2747] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.450167 1023 tablet_service.cc:1468] Processing CreateTablet for tablet aafb7eef813f48fc8295fbb30a29cd0d (DEFAULT_TABLE table=default.Table_129a669a3cfe4cffbe85f526c5d047d5 [id=e3ee2885822c46f7a3347a7d3f99ebb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.453104 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafb7eef813f48fc8295fbb30a29cd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.453969 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6979b2479147ba94842159f0b79c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.454295 2254 ts_tablet_manager.cc:1939] T 9514bbbda0ba4329a18b0a42e635c6e5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:17.455086 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1275) took 1171 ms. Trace:
I20250625 13:52:17.455982 2254 rpcz_store.cc:276] 0625 13:52:16.283626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.283862 (+ 236us) service_pool.cc:225] Handling call
0625 13:52:17.392199 (+1108337us) ts_tablet_manager.cc:1145] Deleting tablet 9514bbbda0ba4329a18b0a42e635c6e5
0625 13:52:17.392206 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.396185 (+ 3979us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.398921 (+ 2736us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.455070 (+ 56149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108108}
I20250625 13:52:17.457878 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86588eb3c34c4536b14f76132bdf3adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.458268 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9b6e6e1f2d41288a958115dc18d4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.462244 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d87f41700df148f4a9be83b417176332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.462863 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d87f41700df148f4a9be83b417176332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.463107 2338 raft_consensus.cc:491] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.463232 2338 raft_consensus.cc:513] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.464321 2338 leader_election.cc:290] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.465063 2254 tablet_replica.cc:331] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:17.465262 2254 raft_consensus.cc:2241] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:17.465466 2254 pending_rounds.cc:70] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:17.465435 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077e8c001cc646559b5268d4a48c0ad6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.465516 2254 pending_rounds.cc:77] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520661704175616 op_type: NO_OP noop_request { }
I20250625 13:52:17.465596 2254 raft_consensus.cc:2887] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:17.465672 2254 raft_consensus.cc:2270] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:17.466552 849 leader_election.cc:343] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.467655 1044 raft_consensus.cc:3058] T b54183cf36b049c7bed30deb1604c59a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.468293 2254 ts_tablet_manager.cc:1905] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:17.472182 851 leader_election.cc:343] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 077e8c001cc646559b5268d4a48c0ad6
I20250625 13:52:17.472293 851 leader_election.cc:304] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.472786 2204 consensus_queue.cc:1035] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:17.473171 2204 raft_consensus.cc:2747] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.475759 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3d5208970197492bbdb0abdb8aefb5b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.476485 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d5208970197492bbdb0abdb8aefb5b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.479707 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 9514bbbda0ba4329a18b0a42e635c6e5 (table default.Table_3462a32d0970423c8538a79dae984471 [id=60ccb9d481c14fd68670e5701ce76b84]) successfully deleted
W20250625 13:52:17.481722 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4f93a7beee14d2892ee387bcc1e76b9 consensus state has no leader
I20250625 13:52:17.483187 2338 raft_consensus.cc:491] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.483299 2338 raft_consensus.cc:513] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.483160 2091 raft_consensus.cc:1215] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.673->[1.674-1.1121] Dedup: 1.699->[1.700-1.1121]
I20250625 13:52:17.483888 2338 leader_election.cc:290] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.484565 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8985f259eb54ffdab29a9f6a2c5ae16" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.485174 849 leader_election.cc:343] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:17.485311 851 leader_election.cc:343] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d8985f259eb54ffdab29a9f6a2c5ae16
I20250625 13:52:17.485419 851 leader_election.cc:304] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.485652 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_736e211328b54c2faf37ec980eae8d82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:17.486024 2338 raft_consensus.cc:2747] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:17.486136 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_736e211328b54c2faf37ec980eae8d82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.487324 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aafb7eef813f48fc8295fbb30a29cd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.487573 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d406206ea8e14121bf43214adfc5afda. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:17.491281 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b493f32299ba42159c594849e0fe39c4 consensus state has no leader
I20250625 13:52:17.498142 2254 ts_tablet_manager.cc:1918] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:17.502475 1048 raft_consensus.cc:3058] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.509740 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a73ef917f48a41e28ee92a70c69e2f5b" } new_table_name: "default.Table_e3e6c0c72937408cb2125f54908cbb0e" modify_external_catalogs: true
I20250625 13:52:17.510126 1170 tablet_bootstrap.cc:492] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:17.510350 1170 ts_tablet_manager.cc:1397] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.003s
I20250625 13:52:17.511049 2254 log.cc:1199] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/37e4bd4aaab143ffa65a668a952ae345
I20250625 13:52:17.513329 882 tablet_service.cc:1468] Processing CreateTablet for tablet d406206ea8e14121bf43214adfc5afda (DEFAULT_TABLE table=default.Table_98ed87f3ecee4add88ff785fe7175ce1 [id=fb0343dcc81d4068b7faab7989c391fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.513753 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d406206ea8e14121bf43214adfc5afda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.512213 2254 ts_tablet_manager.cc:1939] T 37e4bd4aaab143ffa65a668a952ae345 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:17.514197 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1276) took 1230 ms. Trace:
I20250625 13:52:17.515053 2254 rpcz_store.cc:276] 0625 13:52:16.283784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.283838 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:17.465015 (+1181177us) ts_tablet_manager.cc:1145] Deleting tablet 37e4bd4aaab143ffa65a668a952ae345
0625 13:52:17.465023 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.480048 (+ 15025us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.498039 (+ 17991us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.514182 (+ 16143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180773}
I20250625 13:52:17.516526 1171 tablet_bootstrap.cc:492] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:17.516733 1171 ts_tablet_manager.cc:1397] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.153s user 0.002s sys 0.002s
I20250625 13:52:17.517663 1171 raft_consensus.cc:357] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.517953 1171 raft_consensus.cc:383] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.518057 1171 raft_consensus.cc:738] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.518365 1171 consensus_queue.cc:260] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.519039 1171 ts_tablet_manager.cc:1428] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:17.520409 1171 tablet_bootstrap.cc:492] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.517554 2472 raft_consensus.cc:491] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.526330 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 37e4bd4aaab143ffa65a668a952ae345 (table default.Table_3ac6f01366104d75a5bf908bc40c3f4f [id=8a9015750b254a9ea6a48a9510aeddf4]) successfully deleted
I20250625 13:52:17.526311 2472 raft_consensus.cc:513] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.526616 2254 tablet_replica.cc:331] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:17.526801 2254 raft_consensus.cc:2241] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.526990 2254 raft_consensus.cc:2270] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.527068 2472 leader_election.cc:290] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:17.528813 849 consensus_peers.cc:487] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:17.518000 1170 raft_consensus.cc:357] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:17.528329 985 leader_election.cc:343] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 68b8c8aa8036454298bf1e81bab91c01
I20250625 13:52:17.533854 1048 raft_consensus.cc:3058] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.527529 2254 ts_tablet_manager.cc:1905] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.538168 2518 raft_consensus.cc:491] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.538358 2518 raft_consensus.cc:513] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.520260 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 331558137f554be48d24100a9c720f66 (DEFAULT_TABLE table=default.Table_1bf18edd3d52428f9c11971f9df6b8a0 [id=08fa53445f5641e7af67692bc16764db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.522034 2515 tablet.cc:1722] T 32143e2497ee4d1b8c2e7cbfc6160ef4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 13:52:17.541152 886 tablet_service.cc:1468] Processing CreateTablet for tablet 331558137f554be48d24100a9c720f66 (DEFAULT_TABLE table=default.Table_1bf18edd3d52428f9c11971f9df6b8a0 [id=08fa53445f5641e7af67692bc16764db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.541596 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331558137f554be48d24100a9c720f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.534055 1170 raft_consensus.cc:383] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.542671 1170 raft_consensus.cc:738] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.542953 1170 consensus_queue.cc:260] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.543516 1170 ts_tablet_manager.cc:1428] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250625 13:52:17.543844 1170 tablet_bootstrap.cc:492] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:17.545159 1170 tablet_bootstrap.cc:654] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.568881 891 tablet_service.cc:1468] Processing CreateTablet for tablet 4091a34a3d61408780c5b9231a68110c (DEFAULT_TABLE table=default.Table_bc5ae60b26494a4aa4ca38cfdb824db3 [id=e703dc6f1ad943d4b5315ced5da69768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.569684 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 7b59e94e34e645aaa76ea38a55244208 (DEFAULT_TABLE table=default.Table_e0e0b8f7f26d45d4af21d3f2c9219fa5 [id=d3599c91443b4e508bd066d370c05f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.570254 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7b59e94e34e645aaa76ea38a55244208 (DEFAULT_TABLE table=default.Table_e0e0b8f7f26d45d4af21d3f2c9219fa5 [id=d3599c91443b4e508bd066d370c05f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.570787 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 917fc37cf8f9429eba542e2d6ecd2d0c (DEFAULT_TABLE table=default.Table_843c4743b1b346f9a3fa1343436191ac [id=5113ee3dc80346839a305b39789caad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.571090 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b59e94e34e645aaa76ea38a55244208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.571357 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917fc37cf8f9429eba542e2d6ecd2d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.575178 1170 tablet_bootstrap.cc:492] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:17.575294 1170 ts_tablet_manager.cc:1397] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20250625 13:52:17.575731 2472 raft_consensus.cc:491] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.575850 2472 raft_consensus.cc:513] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.576383 2472 leader_election.cc:290] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.576311 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4091a34a3d61408780c5b9231a68110c (DEFAULT_TABLE table=default.Table_bc5ae60b26494a4aa4ca38cfdb824db3 [id=e703dc6f1ad943d4b5315ced5da69768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.576746 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4091a34a3d61408780c5b9231a68110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.581699 2478 raft_consensus.cc:491] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.581871 2478 raft_consensus.cc:513] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.582664 750 tablet_service.cc:1468] Processing CreateTablet for tablet 331558137f554be48d24100a9c720f66 (DEFAULT_TABLE table=default.Table_1bf18edd3d52428f9c11971f9df6b8a0 [id=08fa53445f5641e7af67692bc16764db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.582747 2338 raft_consensus.cc:491] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.582880 2338 raft_consensus.cc:513] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.583707 1166 tablet_bootstrap.cc:492] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.583869 1166 ts_tablet_manager.cc:1397] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.170s user 0.004s sys 0.009s
W20250625 13:52:17.584605 985 leader_election.cc:343] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 60ef50a3624b4c79acc803ff0e2ed364
I20250625 13:52:17.585340 742 tablet_service.cc:1468] Processing CreateTablet for tablet 4091a34a3d61408780c5b9231a68110c (DEFAULT_TABLE table=default.Table_bc5ae60b26494a4aa4ca38cfdb824db3 [id=e703dc6f1ad943d4b5315ced5da69768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.585426 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331558137f554be48d24100a9c720f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.585902 2515 tablet.cc:1722] T 5e280bd329764924a6fd42c591888ed0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:17.586449 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4091a34a3d61408780c5b9231a68110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.588110 2478 leader_election.cc:290] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.588244 745 tablet_service.cc:1468] Processing CreateTablet for tablet 917fc37cf8f9429eba542e2d6ecd2d0c (DEFAULT_TABLE table=default.Table_843c4743b1b346f9a3fa1343436191ac [id=5113ee3dc80346839a305b39789caad9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.588959 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e3ef7f5b474466bbc19fd31c6b6d99c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:17.589435 711 leader_election.cc:343] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.592900 889 tablet_service.cc:1468] Processing CreateTablet for tablet 917fc37cf8f9429eba542e2d6ecd2d0c (DEFAULT_TABLE table=default.Table_843c4743b1b346f9a3fa1343436191ac [id=5113ee3dc80346839a305b39789caad9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:17.593230 983 leader_election.cc:343] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 68b8c8aa8036454298bf1e81bab91c01
I20250625 13:52:17.593325 2338 leader_election.cc:290] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.593327 983 leader_election.cc:304] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.592911 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60ef50a3624b4c79acc803ff0e2ed364" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:17.593744 913 raft_consensus.cc:2466] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:17.593909 2472 raft_consensus.cc:2747] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.594168 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "413afd7b62d24fe8b9eceadd57e29fc8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.576326 1170 raft_consensus.cc:357] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:17.595098 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 3d8ce12926e044ee95291a5e9168277a (table default.Table_050b3df547a443f797e988c8fcb3dc97 [id=17cc7336c1144e4dbe7fe181b32646f5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3d8ce12926e044ee95291a5e9168277a consensus state has no leader
I20250625 13:52:17.595811 1170 raft_consensus.cc:383] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.595887 1170 raft_consensus.cc:738] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:17.596029 851 leader_election.cc:343] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 413afd7b62d24fe8b9eceadd57e29fc8
I20250625 13:52:17.595170 983 leader_election.cc:304] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.596107 1170 consensus_queue.cc:260] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.600040 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331558137f554be48d24100a9c720f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.600380 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e3ef7f5b474466bbc19fd31c6b6d99c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.600953 711 leader_election.cc:343] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.601061 711 leader_election.cc:304] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.601428 2478 raft_consensus.cc:2747] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:17.601765 849 leader_election.cc:343] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.601861 849 leader_election.cc:304] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.602326 2204 raft_consensus.cc:2747] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.605880 2471 raft_consensus.cc:2802] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:17.605960 2471 raft_consensus.cc:491] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:17.606021 2471 raft_consensus.cc:3058] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.607376 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4091a34a3d61408780c5b9231a68110c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.607642 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b59e94e34e645aaa76ea38a55244208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.608461 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1169) took 3429 ms. Trace:
I20250625 13:52:17.608541 1168 rpcz_store.cc:276] 0625 13:52:14.178599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.189691 (+ 11092us) service_pool.cc:225] Handling call
0625 13:52:17.551965 (+3362274us) ts_tablet_manager.cc:1145] Deleting tablet fc9a79f066e84a7b87439ea4516f0287
0625 13:52:17.607072 (+ 55107us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.608443 (+ 1371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3362061}
I20250625 13:52:17.608630 2471 raft_consensus.cc:513] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.609084 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917fc37cf8f9429eba542e2d6ecd2d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.609381 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1170) took 3407 ms. Trace:
I20250625 13:52:17.609450 1168 rpcz_store.cc:276] 0625 13:52:14.201852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.201916 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:17.609275 (+3407359us) ts_tablet_manager.cc:1145] Deleting tablet fc9a79f066e84a7b87439ea4516f0287
0625 13:52:17.609284 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.609366 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3407148}
W20250625 13:52:17.609860 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet fc9a79f066e84a7b87439ea4516f0287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9a79f066e84a7b87439ea4516f0287 already in progress: creating tablet
W20250625 13:52:17.610147 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet fc9a79f066e84a7b87439ea4516f0287 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc9a79f066e84a7b87439ea4516f0287 already in progress: creating tablet
I20250625 13:52:17.610409 1168 tablet_replica.cc:331] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:17.610605 1168 raft_consensus.cc:2241] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.610713 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 32143e2497ee4d1b8c2e7cbfc6160ef4 (table default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916]). Expected version 5 got 0
I20250625 13:52:17.610797 1168 raft_consensus.cc:2270] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.611122 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_add52e39ed4c414c9c9e930208ab5c10 [id=d7cbe338b2554a6e99b07325632aaa33] alter complete (version 4)
I20250625 13:52:17.612107 2472 raft_consensus.cc:491] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.612308 2478 raft_consensus.cc:491] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.612285 2472 raft_consensus.cc:513] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.612437 2478 raft_consensus.cc:513] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.613066 2472 leader_election.cc:290] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.613482 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf7fab53cae49908d2952cdaa07943a" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:17.613659 915 raft_consensus.cc:2466] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:17.614079 983 leader_election.cc:304] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:17.614166 985 leader_election.cc:343] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 60ef50a3624b4c79acc803ff0e2ed364
I20250625 13:52:17.614279 1168 ts_tablet_manager.cc:1905] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.614343 2472 raft_consensus.cc:2802] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:17.614427 2472 raft_consensus.cc:491] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:17.614507 2472 raft_consensus.cc:3058] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.614751 757 tablet_service.cc:1468] Processing CreateTablet for tablet 7b59e94e34e645aaa76ea38a55244208 (DEFAULT_TABLE table=default.Table_e0e0b8f7f26d45d4af21d3f2c9219fa5 [id=d3599c91443b4e508bd066d370c05f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:17.614925 2471 leader_election.cc:290] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.615114 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b59e94e34e645aaa76ea38a55244208. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:17.615221 985 leader_election.cc:343] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: dcf7fab53cae49908d2952cdaa07943a
I20250625 13:52:17.617781 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa1c198dd6541809843244c7cc18194" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.618163 2472 raft_consensus.cc:513] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:17.618323 711 leader_election.cc:343] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.619452 2472 leader_election.cc:290] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:17.619767 985 leader_election.cc:343] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: dcf7fab53cae49908d2952cdaa07943a
I20250625 13:52:17.621733 878 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c75deb64c74f0282efeed827659ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.621767 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a57268a18f4f00a39f8360f9c90737 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.621999 879 tablet_service.cc:1515] Processing DeleteTablet for tablet b7d888e628a94f939ecf2571e60b1948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.625608 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60ef50a3624b4c79acc803ff0e2ed364" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:17.625788 915 raft_consensus.cc:3058] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.626978 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_603f8675b541481380889a4e72b78dbc" } modify_external_catalogs: true
I20250625 13:52:17.627166 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5a6a094978314b53847a823ba43ac189" } modify_external_catalogs: true
I20250625 13:52:17.627199 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_603f8675b541481380889a4e72b78dbc" } modify_external_catalogs: true
I20250625 13:52:17.627269 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5a6a094978314b53847a823ba43ac189" } modify_external_catalogs: true
I20250625 13:52:17.626978 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e535d4720af347929711c7e22dd487ed" } new_table_name: "default.Table_3c034ee7d89b44bc8bd5cc49c7ac6c8d" modify_external_catalogs: true
I20250625 13:52:17.628602 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 917fc37cf8f9429eba542e2d6ecd2d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:17.629496 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3ddbb86e81474e9ca79571a9c0372b74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.629928 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ddbb86e81474e9ca79571a9c0372b74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.631011 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ed99b177ce51470a970fe7a1d9e6e25e" } modify_external_catalogs: true
I20250625 13:52:17.631124 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ed99b177ce51470a970fe7a1d9e6e25e" } modify_external_catalogs: true
I20250625 13:52:17.632503 1166 raft_consensus.cc:357] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.632678 1166 raft_consensus.cc:383] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.632756 1166 raft_consensus.cc:738] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.632952 1166 consensus_queue.cc:260] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.634781 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_32b73fb2c3b84d13b81857f6d4a460e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.635218 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32b73fb2c3b84d13b81857f6d4a460e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.635605 2478 leader_election.cc:290] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.636770 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cf58706a138644a49767bbb2201a120e" } modify_external_catalogs: true
I20250625 13:52:17.638155 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cf58706a138644a49767bbb2201a120e" } modify_external_catalogs: true
I20250625 13:52:17.641955 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf7fab53cae49908d2952cdaa07943a" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:17.642134 913 raft_consensus.cc:3058] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:17.643384 2518 leader_election.cc:290] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.646044 915 raft_consensus.cc:2466] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:17.646468 851 leader_election.cc:343] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f3c75deb64c74f0282efeed827659ec2
I20250625 13:52:17.646766 983 leader_election.cc:304] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.646972 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c75deb64c74f0282efeed827659ec2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:17.647432 2472 raft_consensus.cc:2802] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:52:17.647432 849 leader_election.cc:343] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.648197 2472 raft_consensus.cc:695] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:17.648515 2472 consensus_queue.cc:237] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.651628 1171 tablet_bootstrap.cc:654] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.651883 849 leader_election.cc:304] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.653278 2518 raft_consensus.cc:2747] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.665360 592 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
I20250625 13:52:17.665992 879 tablet_service.cc:1515] Processing DeleteTablet for tablet d26820eaba98401da3a8b5348be21fc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bf2f307ad7c4a6a856ddfb6d6fa2b57) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.666083 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 413afd7b62d24fe8b9eceadd57e29fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.666249 2478 raft_consensus.cc:491] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.666416 2478 raft_consensus.cc:513] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.667047 2478 leader_election.cc:290] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.667730 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:17.668040 742 tablet_service.cc:1515] Processing DeleteTablet for tablet eafc402d809245d189f3977162506ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:17.669256 711 leader_election.cc:343] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.669703 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7e2035c1716744f7837845ba75f8b094" } modify_external_catalogs: true
I20250625 13:52:17.669895 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7e2035c1716744f7837845ba75f8b094" } modify_external_catalogs: true
I20250625 13:52:17.670324 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:17.653164 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa1c198dd6541809843244c7cc18194" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:17.676748 711 leader_election.cc:343] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.676872 711 leader_election.cc:304] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.677323 2478 raft_consensus.cc:2747] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.683332 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 70959ec681524518a7ec2c9ed4016ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.595703 2254 ts_tablet_manager.cc:1918] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:17.688225 913 raft_consensus.cc:2466] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:17.690037 1166 ts_tablet_manager.cc:1428] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.106s user 0.005s sys 0.002s
I20250625 13:52:17.695523 1166 tablet_bootstrap.cc:492] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.699743 1166 tablet_bootstrap.cc:654] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.701362 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:17.726403 1171 tablet_bootstrap.cc:492] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:17.726583 1171 ts_tablet_manager.cc:1397] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.206s user 0.007s sys 0.011s
I20250625 13:52:17.727547 1171 raft_consensus.cc:357] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.727756 1171 raft_consensus.cc:383] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.727821 1171 raft_consensus.cc:738] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.728032 1171 consensus_queue.cc:260] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.728626 1171 ts_tablet_manager.cc:1428] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:17.735816 1168 ts_tablet_manager.cc:1918] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.735980 1168 log.cc:1199] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/8063122ea7b24b6790f5c3cc884e09be
I20250625 13:52:17.736304 1168 ts_tablet_manager.cc:1939] T 8063122ea7b24b6790f5c3cc884e09be P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:17.742205 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1172) took 3533 ms. Trace:
I20250625 13:52:17.742308 1168 rpcz_store.cc:276] 0625 13:52:14.209188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.209240 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:17.610360 (+3401120us) ts_tablet_manager.cc:1145] Deleting tablet 8063122ea7b24b6790f5c3cc884e09be
0625 13:52:17.610377 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.732333 (+121956us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.735695 (+ 3362us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.742185 (+ 6490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3400929}
I20250625 13:52:17.742610 1171 tablet_bootstrap.cc:492] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.743003 1168 tablet_replica.cc:331] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:17.743201 1168 raft_consensus.cc:2241] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:17.743175 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 8063122ea7b24b6790f5c3cc884e09be (table default.Table_1ffafeb50d654290963bf41dfeb57f55 [id=75f0bde5d60641f8afe279bd82d16ae1]) successfully deleted
I20250625 13:52:17.743614 1168 raft_consensus.cc:2270] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.744110 1168 ts_tablet_manager.cc:1905] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.744263 1171 tablet_bootstrap.cc:654] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.688759 983 leader_election.cc:304] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.753324 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.753938 711 leader_election.cc:343] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.754097 711 leader_election.cc:304] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.754768 2254 log.cc:1199] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/ab7202a8761444a0a731993d890fcdeb
I20250625 13:52:17.755168 2254 ts_tablet_manager.cc:1939] T ab7202a8761444a0a731993d890fcdeb P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:17.755996 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1277) took 1409 ms. Trace:
I20250625 13:52:17.756574 1166 tablet_bootstrap.cc:492] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.756832 1166 ts_tablet_manager.cc:1397] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20250625 13:52:17.757664 1166 raft_consensus.cc:357] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.757843 1166 raft_consensus.cc:383] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.757895 1166 raft_consensus.cc:738] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.758078 1166 consensus_queue.cc:260] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.758641 1166 ts_tablet_manager.cc:1428] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:17.766537 2548 raft_consensus.cc:2747] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.772509 1166 tablet_bootstrap.cc:492] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.756078 2254 rpcz_store.cc:276] 0625 13:52:16.346968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.347019 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:17.526567 (+1179548us) ts_tablet_manager.cc:1145] Deleting tablet ab7202a8761444a0a731993d890fcdeb
0625 13:52:17.526577 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.553014 (+ 26437us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.595610 (+ 42596us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.755978 (+160368us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179391}
I20250625 13:52:17.773962 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet ab7202a8761444a0a731993d890fcdeb (table default.Table_753dc763991a4d2c900b8f1dc5e6138a [id=22b45bb056454aba99f356fbc831479a]) successfully deleted
I20250625 13:52:17.778028 1166 tablet_bootstrap.cc:654] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.773916 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1278) took 1420 ms. Trace:
I20250625 13:52:17.777622 2471 raft_consensus.cc:2802] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:17.779222 1170 ts_tablet_manager.cc:1428] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.204s user 0.000s sys 0.002s
I20250625 13:52:17.784310 2471 raft_consensus.cc:695] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:17.784785 1170 tablet_bootstrap.cc:492] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:17.785627 2471 consensus_queue.cc:237] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:17.786209 849 consensus_peers.cc:487] T 423ca098138c476f861a19b44554fd24 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:17.784212 2254 rpcz_store.cc:276] 0625 13:52:16.353423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.353481 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:17.773771 (+1420290us) ts_tablet_manager.cc:1145] Deleting tablet 23fa1b7e7c13444fb8ee07807dabd813
0625 13:52:17.773783 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.773897 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420100}
W20250625 13:52:17.790725 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 23fa1b7e7c13444fb8ee07807dabd813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23fa1b7e7c13444fb8ee07807dabd813 already in progress: creating tablet
I20250625 13:52:17.793578 2518 raft_consensus.cc:491] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c1aa07ab6bc4b0089e46951240a37c9)
I20250625 13:52:17.793156 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1279) took 1373 ms. Trace:
I20250625 13:52:17.793812 2518 raft_consensus.cc:513] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:17.795578 849 leader_election.cc:343] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Not found: Tablet not found: 3f56a8a6cb964d3a979e2f1d944eeddc
I20250625 13:52:17.795883 2548 raft_consensus.cc:491] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.794090 2254 rpcz_store.cc:276] 0625 13:52:16.420106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.420277 (+ 171us) service_pool.cc:225] Handling call
0625 13:52:17.791957 (+1371680us) ts_tablet_manager.cc:1145] Deleting tablet 882152a3a5a2458d803ba7f75b31bc30
0625 13:52:17.791970 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.792904 (+ 934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336417}
I20250625 13:52:17.796116 2548 raft_consensus.cc:513] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:17.796646 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 882152a3a5a2458d803ba7f75b31bc30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 882152a3a5a2458d803ba7f75b31bc30 already in progress: creating tablet
I20250625 13:52:17.797646 2548 leader_election.cc:290] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:17.797778 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:17.797988 2523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85834f366fa42c294c9290af21ea111. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:17.798219 2523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: FOLLOWER
W20250625 13:52:17.798295 711 leader_election.cc:343] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.798759 2091 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48780 (request call id 0) took 8578 ms. Trace:
I20250625 13:52:17.801951 1166 tablet_bootstrap.cc:492] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.802121 1166 ts_tablet_manager.cc:1397] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20250625 13:52:17.802837 1166 raft_consensus.cc:357] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:17.807426 1166 raft_consensus.cc:383] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.807544 1166 raft_consensus.cc:738] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.807868 1166 consensus_queue.cc:260] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:17.798576 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.809695 711 leader_election.cc:343] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.809808 711 leader_election.cc:304] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:17.811915 2548 raft_consensus.cc:2747] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:17.817232 1166 ts_tablet_manager.cc:1428] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250625 13:52:17.817646 1166 tablet_bootstrap.cc:492] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:17.819275 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 130561us
I20250625 13:52:17.827353 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1280) took 1356 ms. Trace:
I20250625 13:52:17.827637 2254 rpcz_store.cc:276] 0625 13:52:16.470883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.470935 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:17.812595 (+1341660us) ts_tablet_manager.cc:1145] Deleting tablet e38a85c299f04d60a3c465571fd33971
0625 13:52:17.818071 (+ 5476us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.827276 (+ 9205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341102}
I20250625 13:52:17.829947 1170 tablet_bootstrap.cc:654] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.830724 1166 tablet_bootstrap.cc:654] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.831820 2338 raft_consensus.cc:491] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:17.831961 2338 raft_consensus.cc:513] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:17.832348 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet e38a85c299f04d60a3c465571fd33971 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e38a85c299f04d60a3c465571fd33971 already in progress: creating tablet
I20250625 13:52:17.833413 2338 leader_election.cc:290] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:17.833581 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7d888e628a94f939ecf2571e60b1948" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:17.834175 849 leader_election.cc:343] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:17.835042 2254 tablet_replica.cc:331] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:17.848680 2254 raft_consensus.cc:2241] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:17.849112 2254 pending_rounds.cc:70] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:17.849188 2254 pending_rounds.cc:77] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520666211602432 op_type: NO_OP noop_request { }
I20250625 13:52:17.849361 2254 raft_consensus.cc:2887] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:17.849499 2254 raft_consensus.cc:2270] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.851256 2254 ts_tablet_manager.cc:1905] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:17.858922 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdb955e3410945f898639235cd25bad7 consensus state has no leader
I20250625 13:52:17.859277 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 14087755cd5744958499f32fd77b655b
I20250625 13:52:17.860447 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ed3c177bb6dd488aa54bee2a447016ff (table default.Table_c4f1a08649504a7db750c88b65ff985c [id=4631cf687a9c478ba42e9ea3375ca639]). Expected version 1 got 0
I20250625 13:52:17.860654 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ef2b08fba37f4a38880c018ec2d869c8 (table default.Table_3a4e5f10c2ec453a82fb97800a0d37f7 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]). Expected version 2 got 0
I20250625 13:52:17.861338 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9d0e64e523c24f2ca751d55bb49a7213"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.861860 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d0e64e523c24f2ca751d55bb49a7213 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.862185 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ca0d7155da664e88b759e29a7c5d554b" } new_table_name: "default.Table_08c064ca59a14ff897cf0347dd79584c" modify_external_catalogs: true
I20250625 13:52:17.862349 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_051d5935913847be9d19460f80f73e22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.862953 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_051d5935913847be9d19460f80f73e22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.867969 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7d7b671a935b43a6a80a37d38e57f230
I20250625 13:52:17.869117 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d7b671a935b43a6a80a37d38e57f230 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:17.869748 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0a13c34b24244d58daa0d67589b5100 consensus state has no leader
I20250625 13:52:17.871505 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d7b671a935b43a6a80a37d38e57f230 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.871835 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 14087755cd5744958499f32fd77b655b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.872632 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 14087755cd5744958499f32fd77b655b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.872949 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 14087755cd5744958499f32fd77b655b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.872996 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d7b671a935b43a6a80a37d38e57f230 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:17.873124 849 consensus_peers.cc:487] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:17.874959 2518 leader_election.cc:290] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:17.874975 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ed3c177bb6dd488aa54bee2a447016ff (table default.Table_c4f1a08649504a7db750c88b65ff985c [id=4631cf687a9c478ba42e9ea3375ca639]). Expected version 1 got 0
I20250625 13:52:17.875149 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
I20250625 13:52:17.875341 597 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 71cf3b95139245a4ba11b6bcaa3ab3f4 (table default.Table_4ce1c9ba4cf14145821fc2430363bf62 [id=b3427651d7264e36922da55c8d267546]). Expected version 1 got 0
W20250625 13:52:17.875425 851 leader_election.cc:343] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b7d888e628a94f939ecf2571e60b1948
W20250625 13:52:17.876032 849 consensus_peers.cc:487] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:17.876389 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_77d42655ef0349128a8518e3892d4532" } new_table_name: "default.Table_c183e5f797384e9993de44fd16071209" modify_external_catalogs: true
I20250625 13:52:17.876783 1168 ts_tablet_manager.cc:1918] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:17.876894 1168 log.cc:1199] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/39ac79102eab46688e767faed6d6f114
I20250625 13:52:17.877295 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e5b4162d2ed49358c920ebbce0cfa5c" } new_table_name: "default.Table_40bba3ccaa054c1cabdb0e640f19575f" modify_external_catalogs: true
I20250625 13:52:17.875530 851 leader_election.cc:304] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.877897 2204 raft_consensus.cc:2747] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:17.879459 851 leader_election.cc:343] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3f56a8a6cb964d3a979e2f1d944eeddc
I20250625 13:52:17.879568 851 leader_election.cc:304] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:17.880002 2518 raft_consensus.cc:2747] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:17.880268 742 tablet_service.cc:1515] Processing DeleteTablet for tablet e3939ef22ea344328721dfba67d933d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.881194 1168 ts_tablet_manager.cc:1939] T 39ac79102eab46688e767faed6d6f114 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:17.881572 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1183) took 3530 ms. Trace:
I20250625 13:52:17.881686 1168 rpcz_store.cc:276] 0625 13:52:14.351599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.351649 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:17.742617 (+3390968us) ts_tablet_manager.cc:1145] Deleting tablet 39ac79102eab46688e767faed6d6f114
0625 13:52:17.742959 (+ 342us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.749412 (+ 6453us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.876645 (+127233us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.881557 (+ 4912us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3390775}
I20250625 13:52:17.882153 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 39ac79102eab46688e767faed6d6f114 (table default.Table_f26987562c25452388d457eb3b600595 [id=62553d70f8ee470f944206ddc8cc2089]) successfully deleted
I20250625 13:52:17.882527 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1184) took 3524 ms. Trace:
I20250625 13:52:17.882644 1168 rpcz_store.cc:276] 0625 13:52:14.357607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.357657 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:17.882419 (+3524762us) ts_tablet_manager.cc:1145] Deleting tablet d0174345facc4216b6ddeb8f8c73790d
0625 13:52:17.882428 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.882512 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3524586}
W20250625 13:52:17.883077 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d0174345facc4216b6ddeb8f8c73790d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0174345facc4216b6ddeb8f8c73790d already in progress: creating tablet
I20250625 13:52:17.883482 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1181) took 3538 ms. Trace:
I20250625 13:52:17.862954 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6cabec757d6749df9a8aae136342febf
I20250625 13:52:17.883596 1168 rpcz_store.cc:276] 0625 13:52:14.345294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.345370 (+ 76us) service_pool.cc:225] Handling call
0625 13:52:17.883394 (+3538024us) ts_tablet_manager.cc:1145] Deleting tablet 3562e8b8731a4d76bb1a36189f957e07
0625 13:52:17.883402 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.883469 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3493336}
W20250625 13:52:17.884074 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 3562e8b8731a4d76bb1a36189f957e07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3562e8b8731a4d76bb1a36189f957e07 already in progress: creating tablet
I20250625 13:52:17.884393 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1182) took 3535 ms. Trace:
I20250625 13:52:17.884822 1168 rpcz_store.cc:276] 0625 13:52:14.349232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.349284 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:17.884310 (+3535026us) ts_tablet_manager.cc:1145] Deleting tablet 8f88091205b944b386094e50cf92c36b
0625 13:52:17.884318 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.884381 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3470630}
I20250625 13:52:17.884938 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cabec757d6749df9a8aae136342febf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:17.885218 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 8f88091205b944b386094e50cf92c36b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f88091205b944b386094e50cf92c36b already in progress: creating tablet
I20250625 13:52:17.885978 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1195) took 3430 ms. Trace:
I20250625 13:52:17.886106 1168 rpcz_store.cc:276] 0625 13:52:14.455244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.455331 (+ 87us) service_pool.cc:225] Handling call
0625 13:52:17.885499 (+3430168us) ts_tablet_manager.cc:1145] Deleting tablet 529c3bf8e72f46948cfb864445ff51f3
0625 13:52:17.885511 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.885963 (+ 452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3429992}
I20250625 13:52:17.879841 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe4046742b34f1ab0b1610b5d6ee01e with delete_type TABLET_DATA_DELETED (replaced by tablet 066db01f791542a38cfb6e408c4836ed) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.884306 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cabec757d6749df9a8aae136342febf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.887123 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cabec757d6749df9a8aae136342febf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.887321 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d15a9f7239f84b448320dfae951349cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:17.887851 1170 tablet_bootstrap.cc:492] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:17.887904 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d15a9f7239f84b448320dfae951349cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.888096 1170 ts_tablet_manager.cc:1397] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.002s
I20250625 13:52:17.888861 1170 raft_consensus.cc:357] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.889106 1170 raft_consensus.cc:383] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.889230 1170 raft_consensus.cc:738] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.889540 1170 consensus_queue.cc:260] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.889752 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1df4d80b0f28482e902400731cf43a7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.890270 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1df4d80b0f28482e902400731cf43a7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:17.890897 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 529c3bf8e72f46948cfb864445ff51f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 529c3bf8e72f46948cfb864445ff51f3 already in progress: creating tablet
I20250625 13:52:17.892983 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33015
I20250625 13:52:17.894861 1170 ts_tablet_manager.cc:1428] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 13:52:17.899379 1168 tablet_replica.cc:331] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:17.899624 1168 raft_consensus.cc:2241] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.899830 1168 raft_consensus.cc:2270] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.900693 1170 tablet_bootstrap.cc:492] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:17.901749 1166 tablet_bootstrap.cc:492] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.901940 1166 ts_tablet_manager.cc:1397] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.002s
I20250625 13:52:17.902148 1168 ts_tablet_manager.cc:1905] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.902474 1170 tablet_bootstrap.cc:654] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.902719 1166 raft_consensus.cc:357] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.902966 1166 raft_consensus.cc:383] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.903067 1166 raft_consensus.cc:738] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.903373 1166 consensus_queue.cc:260] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.906526 1166 ts_tablet_manager.cc:1428] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:17.906888 1166 tablet_bootstrap.cc:492] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.908574 1166 tablet_bootstrap.cc:654] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.908653 1171 tablet_bootstrap.cc:492] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:17.911435 1171 ts_tablet_manager.cc:1397] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.169s user 0.029s sys 0.033s
I20250625 13:52:17.912232 1171 raft_consensus.cc:357] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.912542 1171 raft_consensus.cc:383] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.912647 1171 raft_consensus.cc:738] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.912884 1171 consensus_queue.cc:260] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.919763 1171 ts_tablet_manager.cc:1428] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:52:17.911132 1165 ts_tablet_manager.cc:1918] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.925134 1165 log.cc:1199] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/d9462da7574a46839763b1c388102e60
I20250625 13:52:17.925592 1165 ts_tablet_manager.cc:1939] T d9462da7574a46839763b1c388102e60 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:17.926679 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet d9462da7574a46839763b1c388102e60 (table default.Table_866deeb74a8743f0a579513bf580c9e9 [id=ce6cacfdab30448687a65de93031ab3e]) successfully deleted
I20250625 13:52:17.799408 2091 rpcz_store.cc:276] 0625 13:52:09.220011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:09.220064 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:17.456081 (+8236017us) raft_consensus.cc:1469] Updating replica for 448 ops
0625 13:52:17.483991 (+ 27910us) raft_consensus.cc:1532] Early marking committed up to index 699
0625 13:52:17.542062 (+ 58071us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:52:17.542070 (+ 8us) write_op.cc:454] Released schema lock
0625 13:52:17.545865 (+ 3795us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:52:17.545871 (+ 6us) write_op.cc:454] Released schema lock
0625 13:52:17.549777 (+ 3906us) write_op.cc:489] Releasing partition, row and schema locks
0625 13:52:17.549783 (+ 6us) write_op.cc:454] Released schema lock
0625 13:52:17.550349 (+ 566us) raft_consensus.cc:1537] Triggering prepare for 422 ops
0625 13:52:17.550700 (+ 351us) write_op.cc:270] Start()
0625 13:52:17.550745 (+ 45us) write_op.cc:276] Timestamp: P: 1750859521970680 usec, L: 0
0625 13:52:17.551008 (+ 263us) write_op.cc:270] Start()
0625 13:52:17.551030 (+ 22us) write_op.cc:276] Timestamp: P: 1750859521976150 usec, L: 0
0625 13:52:17.551168 (+ 138us) write_op.cc:270] Start()
0625 13:52:17.551186 (+ 18us) write_op.cc:276] Timestamp: P: 1750859521996868 usec, L: 0
0625 13:52:17.551379 (+ 193us) write_op.cc:270] Start()
0625 13:52:17.551399 (+ 20us) write_op.cc:276] Timestamp: P: 1750859521997543 usec, L: 0
0625 13:52:17.551570 (+ 171us) write_op.cc:270] Start()
0625 13:52:17.551589 (+ 19us) write_op.cc:276] Timestamp: P: 1750859522027557 usec, L: 0
0625 13:52:17.560057 (+ 8468us) write_op.cc:270] Start()
0625 13:52:17.560085 (+ 28us) write_op.cc:276] Timestamp: P: 1750859522035957 usec, L: 0
0625 13:52:17.560235 (+ 150us) write_op.cc:270] Start()
0625 13:52:17.560255 (+ 20us) write_op.cc:276] Timestamp: P: 1750859522058072 usec, L: 0
0625 13:52:17.560413 (+ 158us) write_op.cc:270] Start()
0625 13:52:17.560434 (+ 21us) write_op.cc:276] Timestamp: P: 1750859522080019 usec, L: 0
0625 13:52:17.560662 (+ 228us) write_op.cc:270] Start()
0625 13:52:17.560684 (+ 22us) write_op.cc:276] Timestamp: P: 1750859522085024 usec, L: 0
0625 13:52:17.560843 (+ 159us) write_op.cc:270] Start()
0625 13:52:17.560864 (+ 21us) write_op.cc:276] Timestamp: P: 1750859522094822 usec, L: 0
0625 13:52:17.561014 (+ 150us) write_op.cc:270] Start()
0625 13:52:17.561034 (+ 20us) write_op.cc:276] Timestamp: P: 1750859522116845 usec, L: 0
0625 13:52:17.561164 (+ 130us) write_op.cc:270] Start()
0625 13:52:17.561183 (+ 19us) write_op.cc:276] Timestamp: P: 1750859522123531 usec, L: 0
0625 13:52:17.561317 (+ 134us) write_op.cc:270] Start()
0625 13:52:17.561338 (+ 21us) write_op.cc:276] Timestamp: P: 1750859522152265 usec, L: 0
0625 13:52:17.561514 (+ 176us) write_op.cc:270] Start()
0625 13:52:17.561538 (+ 24us) write_op.cc:276] Timestamp: P: 1750859522153881 usec, L: 0
0625 13:52:17.561692 (+ 154us) write_op.cc:270] Start()
0625 13:52:17.561711 (+ 19us) write_op.cc:276] Timestamp: P: 1750859522168129 usec, L: 0
0625 13:52:17.561873 (+ 162us) write_op.cc:270] Start()
0625 13:52:17.561901 (+ 28us) write_op.cc:276] Timestamp: P: 1750859522176526 usec, L: 0
0625 13:52:17.562073 (+ 172us) write_op.cc:270] Start()
0625 13:52:17.562098 (+ 25us) write_op.cc:276] Timestamp: P: 1750859522188556 usec, L: 0
0625 13:52:17.562245 (+ 147us) write_op.cc:270] Start()
0625 13:52:17.562324 (+ 79us) write_op.cc:276] Timestamp: P: 1750859522196213 usec, L: 0
0625 13:52:17.562534 (+ 210us) write_op.cc:270] Start()
0625 13:52:17.562553 (+ 19us) write_op.cc:276] Timestamp: P: 1750859522218178 usec, L: 0
0625 13:52:17.562665 (+ 112us) write_op.cc:270] Start()
0625 13:52:17.562686 (+ 21us) write_op.cc:276] Timestamp: P: 1750859522231889 usec, L: 0
0625 13:52:17.562821 (+ 135us) write_op.cc:270] Start()
0625 13:52:17.562896 (+ 75us) write_op.cc:276] Timestamp: P: 1750859522243771 usec, L: 0
0625 13:52:17.563847 (+ 951us) write_op.cc:270] Start()
0625 13:52:17.563882 (+ 35us) write_op.cc:276] Timestamp: P: 1750859522260569 usec, L: 0
0625 13:52:17.564093 (+ 211us) write_op.cc:270] Start()
0625 13:52:17.564127 (+ 34us) write_op.cc:276] Timestamp: P: 1750859522276186 usec, L: 0
0625 13:52:17.564305 (+ 178us) write_op.cc:270] Start()
0625 13:52:17.564331 (+ 26us) write_op.cc:276] Timestamp: P: 1750859522279829 usec, L: 0
0625 13:52:17.564536 (+ 205us) write_op.cc:270] Start()
0625 13:52:17.564561 (+ 25us) write_op.cc:276] Timestamp: P: 1750859522287309 usec, L: 0
0625 13:52:17.564700 (+ 139us) write_op.cc:270] Start()
0625 13:52:17.564723 (+ 23us) write_op.cc:276] Timestamp: P: 1750859522314250 usec, L: 0
0625 13:52:17.569018 (+ 4295us) write_op.cc:270] Start()
0625 13:52:17.569046 (+ 28us) write_op.cc:276] Timestamp: P: 1750859522334620 usec, L: 0
0625 13:52:17.569197 (+ 151us) write_op.cc:270] Start()
0625 13:52:17.569218 (+ 21us) write_op.cc:276] Timestamp: P: 1750859522352973 usec, L: 0
0625 13:52:17.569349 (+ 131us) write_op.cc:270] Start()
0625 13:52:17.569368 (+ 19us) write_op.cc:276] Timestamp: P: 1750859522380599 usec, L: 0
0625 13:52:17.569527 (+ 159us) write_op.cc:270] Start()
0625 13:52:17.569552 (+ 25us) write_op.cc:276] Timestamp: P: 1750859522405265 usec, L: 0
0625 13:52:17.569690 (+ 138us) write_op.cc:270] Start()
0625 13:52:17.569713 (+ 23us) write_op.cc:276] Timestamp: P: 1750859522423126 usec, L: 0
0625 13:52:17.569881 (+ 168us) write_op.cc:270] Start()
0625 13:52:17.569900 (+ 19us) write_op.cc:276] Timestamp: P: 1750859522438695 usec, L: 0
0625 13:52:17.570066 (+ 166us) write_op.cc:270] Start()
0625 13:52:17.570086 (+ 20us) write_op.cc:276] Timestamp: P: 1750859522454422 usec, L: 0
0625 13:52:17.570288 (+ 202us) write_op.cc:270] Start()
0625 13:52:17.570333 (+ 45us) write_op.cc:276] Timestamp: P: 1750859522469513 usec, L: 0
0625 13:52:17.570503 (+ 170us) write_op.cc:270] Start()
0625 13:52:17.570526 (+ 23us) write_op.cc:276] Timestamp: P: 1750859522504709 usec, L: 0
0625 13:52:17.577731 (+ 7205us) write_op.cc:270] Start()
0625 13:52:17.577764 (+ 33us) write_op.cc:276] Timestamp: P: 1750859522513523 usec, L: 0
0625 13:52:17.577934 (+ 170us) write_op.cc:270] Start()
0625 13:52:17.577964 (+ 30us) write_op.cc:276] Timestamp: P: 1750859522520991 usec, L: 0
0625 13:52:17.578097 (+ 133us) write_op.cc:270] Start()
0625 13:52:17.578112 (+ 15us) write_op.cc:276] Timestamp: P: 1750859522540411 usec, L: 0
0625 13:52:17.578225 (+ 113us) write_op.cc:270] Start()
0625 13:52:17.578241 (+ 16us) write_op.cc:276] Timestamp: P: 1750859522543980 usec, L: 0
0625 13:52:17.578348 (+ 107us) write_op.cc:270] Start()
0625 13:52:17.578368 (+ 20us) write_op.cc:276] Timestamp: P: 1750859522555418 usec, L: 0
0625 13:52:17.578532 (+ 164us) write_op.cc:270] Start()
0625 13:52:17.578552 (+ 20us) write_op.cc:276] Timestamp: P: 1750859522563228 usec, L: 0
0625 13:52:17.578728 (+ 176us) write_op.cc:270] Start()
0625 13:52:17.578755 (+ 27us) write_op.cc:276] Timestamp: P: 1750859522576993 usec, L: 0
0625 13:52:17.578901 (+ 146us) write_op.cc:270] Start()
0625 13:52:17.578922 (+ 21us) write_op.cc:276] Timestamp: P: 1750859522589967 usec, L: 0
0625 13:52:17.579073 (+ 151us) write_op.cc:270] Start()
0625 13:52:17.579091 (+ 18us) write_op.cc:276] Timestamp: P: 1750859522601108 usec, L: 0
0625 13:52:17.579259 (+ 168us) write_op.cc:270] Start()
0625 13:52:17.579282 (+ 23us) write_op.cc:276] Timestamp: P: 1750859522603682 usec, L: 0
0625 13:52:17.579434 (+ 152us) write_op.cc:270] Start()
0625 13:52:17.579455 (+ 21us) write_op.cc:276] Timestamp: P: 1750859522605861 usec, L: 0
0625 13:52:17.579598 (+ 143us) write_op.cc:270] Start()
0625 13:52:17.579618 (+ 20us) write_op.cc:276] Timestamp: P: 1750859522618449 usec, L: 0
0625 13:52:17.579734 (+ 116us) write_op.cc:270] Start()
0625 13:52:17.579754 (+ 20us) write_op.cc:276] Timestamp: P: 1750859522619476 usec, L: 0
0625 13:52:17.586134 (+ 6380us) write_op.cc:270] Start()
0625 13:52:17.586163 (+ 29us) write_op.cc:276] Timestamp: P: 1750859522639103 usec, L: 0
0625 13:52:17.586350 (+ 187us) write_op.cc:270] Start()
0625 13:52:17.586402 (+ 52us) write_op.cc:276] Timestamp: P: 1750859522644945 usec, L: 0
0625 13:52:17.586577 (+ 175us) write_op.cc:270] Start()
0625 13:52:17.586599 (+ 22us) write_op.cc:276] Timestamp: P: 1750859522649527 usec, L: 0
0625 13:52:17.586735 (+ 136us) write_op.cc:270] Start()
0625 13:52:17.586757 (+ 22us) write_op.cc:276] Timestamp: P: 1750859522653065 usec, L: 0
0625 13:52:17.586908 (+ 151us) write_op.cc:270] Start()
0625 13:52:17.586934 (+ 26us) write_op.cc:276] Timestamp: P: 1750859522656477 usec, L: 0
0625 13:52:17.587075 (+ 141us) write_op.cc:270] Start()
0625 13:52:17.587092 (+ 17us) write_op.cc:276] Timestamp: P: 1750859522987747 usec, L: 0
0625 13:52:17.587201 (+ 109us) write_op.cc:270] Start()
0625 13:52:17.587236 (+ 35us) write_op.cc:276] Timestamp: P: 1750859522995500 usec, L: 0
0625 13:52:17.587388 (+ 152us) write_op.cc:270] Start()
0625 13:52:17.587406 (+ 18us) write_op.cc:276] Timestamp: P: 1750859523016582 usec, L: 0
0625 13:52:17.587558 (+ 152us) write_op.cc:270] Start()
0625 13:52:17.587574 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523028208 usec, L: 0
0625 13:52:17.587674 (+ 100us) write_op.cc:270] Start()
0625 13:52:17.587688 (+ 14us) write_op.cc:276] Timestamp: P: 1750859523028960 usec, L: 0
0625 13:52:17.587893 (+ 205us) write_op.cc:270] Start()
0625 13:52:17.587918 (+ 25us) write_op.cc:276] Timestamp: P: 1750859523029596 usec, L: 0
0625 13:52:17.588066 (+ 148us) write_op.cc:270] Start()
0625 13:52:17.588086 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523300160 usec, L: 0
0625 13:52:17.588217 (+ 131us) write_op.cc:270] Start()
0625 13:52:17.588239 (+ 22us) write_op.cc:276] Timestamp: P: 1750859523301994 usec, L: 0
0625 13:52:17.588378 (+ 139us) write_op.cc:270] Start()
0625 13:52:17.588409 (+ 31us) write_op.cc:276] Timestamp: P: 1750859523315595 usec, L: 0
0625 13:52:17.588564 (+ 155us) write_op.cc:270] Start()
0625 13:52:17.588585 (+ 21us) write_op.cc:276] Timestamp: P: 1750859523317015 usec, L: 0
0625 13:52:17.588739 (+ 154us) write_op.cc:270] Start()
0625 13:52:17.588758 (+ 19us) write_op.cc:276] Timestamp: P: 1750859523318785 usec, L: 0
0625 13:52:17.588911 (+ 153us) write_op.cc:270] Start()
0625 13:52:17.588969 (+ 58us) write_op.cc:276] Timestamp: P: 1750859523319985 usec, L: 0
0625 13:52:17.589106 (+ 137us) write_op.cc:270] Start()
0625 13:52:17.589126 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523321105 usec, L: 0
0625 13:52:17.589285 (+ 159us) write_op.cc:270] Start()
0625 13:52:17.589306 (+ 21us) write_op.cc:276] Timestamp: P: 1750859523322707 usec, L: 0
0625 13:52:17.589456 (+ 150us) write_op.cc:270] Start()
0625 13:52:17.589490 (+ 34us) write_op.cc:276] Timestamp: P: 1750859523324079 usec, L: 0
0625 13:52:17.589658 (+ 168us) write_op.cc:270] Start()
0625 13:52:17.589679 (+ 21us) write_op.cc:276] Timestamp: P: 1750859523387376 usec, L: 0
0625 13:52:17.589829 (+ 150us) write_op.cc:270] Start()
0625 13:52:17.589849 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523388104 usec, L: 0
0625 13:52:17.589966 (+ 117us) write_op.cc:270] Start()
0625 13:52:17.589986 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523389466 usec, L: 0
0625 13:52:17.590152 (+ 166us) write_op.cc:270] Start()
0625 13:52:17.590178 (+ 26us) write_op.cc:276] Timestamp: P: 1750859523390214 usec, L: 0
0625 13:52:17.590313 (+ 135us) write_op.cc:270] Start()
0625 13:52:17.590333 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523393301 usec, L: 0
0625 13:52:17.590478 (+ 145us) write_op.cc:270] Start()
0625 13:52:17.590504 (+ 26us) write_op.cc:276] Timestamp: P: 1750859523395081 usec, L: 0
0625 13:52:17.590606 (+ 102us) write_op.cc:270] Start()
0625 13:52:17.590622 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523418059 usec, L: 0
0625 13:52:17.590740 (+ 118us) write_op.cc:270] Start()
0625 13:52:17.590756 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523418682 usec, L: 0
0625 13:52:17.590854 (+ 98us) write_op.cc:270] Start()
0625 13:52:17.590869 (+ 15us) write_op.cc:276] Timestamp: P: 1750859523419212 usec, L: 0
0625 13:52:17.590963 (+ 94us) write_op.cc:270] Start()
0625 13:52:17.590976 (+ 13us) write_op.cc:276] Timestamp: P: 1750859523421283 usec, L: 0
0625 13:52:17.591233 (+ 257us) write_op.cc:270] Start()
0625 13:52:17.591295 (+ 62us) write_op.cc:276] Timestamp: P: 1750859523421893 usec, L: 0
0625 13:52:17.591478 (+ 183us) write_op.cc:270] Start()
0625 13:52:17.591495 (+ 17us) write_op.cc:276] Timestamp: P: 1750859523422426 usec, L: 0
0625 13:52:17.591595 (+ 100us) write_op.cc:270] Start()
0625 13:52:17.591611 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523514710 usec, L: 0
0625 13:52:17.591737 (+ 126us) write_op.cc:270] Start()
0625 13:52:17.591753 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523515466 usec, L: 0
0625 13:52:17.591853 (+ 100us) write_op.cc:270] Start()
0625 13:52:17.591869 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523547090 usec, L: 0
0625 13:52:17.591971 (+ 102us) write_op.cc:270] Start()
0625 13:52:17.591987 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523547908 usec, L: 0
0625 13:52:17.592113 (+ 126us) write_op.cc:270] Start()
0625 13:52:17.592131 (+ 18us) write_op.cc:276] Timestamp: P: 1750859523548500 usec, L: 0
0625 13:52:17.592308 (+ 177us) write_op.cc:270] Start()
0625 13:52:17.592325 (+ 17us) write_op.cc:276] Timestamp: P: 1750859523549080 usec, L: 0
0625 13:52:17.592451 (+ 126us) write_op.cc:270] Start()
0625 13:52:17.592477 (+ 26us) write_op.cc:276] Timestamp: P: 1750859523549592 usec, L: 0
0625 13:52:17.592597 (+ 120us) write_op.cc:270] Start()
0625 13:52:17.592615 (+ 18us) write_op.cc:276] Timestamp: P: 1750859523575322 usec, L: 0
0625 13:52:17.593003 (+ 388us) write_op.cc:270] Start()
0625 13:52:17.593026 (+ 23us) write_op.cc:276] Timestamp: P: 1750859523592088 usec, L: 0
0625 13:52:17.593184 (+ 158us) write_op.cc:270] Start()
0625 13:52:17.593204 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523628449 usec, L: 0
0625 13:52:17.593336 (+ 132us) write_op.cc:270] Start()
0625 13:52:17.593359 (+ 23us) write_op.cc:276] Timestamp: P: 1750859523644845 usec, L: 0
0625 13:52:17.593513 (+ 154us) write_op.cc:270] Start()
0625 13:52:17.593568 (+ 55us) write_op.cc:276] Timestamp: P: 1750859523645479 usec, L: 0
0625 13:52:17.593714 (+ 146us) write_op.cc:270] Start()
0625 13:52:17.593734 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523646006 usec, L: 0
0625 13:52:17.593867 (+ 133us) write_op.cc:270] Start()
0625 13:52:17.593886 (+ 19us) write_op.cc:276] Timestamp: P: 1750859523646589 usec, L: 0
0625 13:52:17.594010 (+ 124us) write_op.cc:270] Start()
0625 13:52:17.594028 (+ 18us) write_op.cc:276] Timestamp: P: 1750859523651509 usec, L: 0
0625 13:52:17.594166 (+ 138us) write_op.cc:270] Start()
0625 13:52:17.594188 (+ 22us) write_op.cc:276] Timestamp: P: 1750859523653710 usec, L: 0
0625 13:52:17.594318 (+ 130us) write_op.cc:270] Start()
0625 13:52:17.594336 (+ 18us) write_op.cc:276] Timestamp: P: 1750859523657486 usec, L: 0
0625 13:52:17.594513 (+ 177us) write_op.cc:270] Start()
0625 13:52:17.594532 (+ 19us) write_op.cc:276] Timestamp: P: 1750859523658723 usec, L: 0
0625 13:52:17.594665 (+ 133us) write_op.cc:270] Start()
0625 13:52:17.594687 (+ 22us) write_op.cc:276] Timestamp: P: 1750859523663934 usec, L: 0
0625 13:52:17.594864 (+ 177us) write_op.cc:270] Start()
0625 13:52:17.594885 (+ 21us) write_op.cc:276] Timestamp: P: 1750859523763583 usec, L: 0
0625 13:52:17.595851 (+ 966us) write_op.cc:270] Start()
0625 13:52:17.595877 (+ 26us) write_op.cc:276] Timestamp: P: 1750859523784386 usec, L: 0
0625 13:52:17.596037 (+ 160us) write_op.cc:270] Start()
0625 13:52:17.596061 (+ 24us) write_op.cc:276] Timestamp: P: 1750859523784978 usec, L: 0
0625 13:52:17.596204 (+ 143us) write_op.cc:270] Start()
0625 13:52:17.596224 (+ 20us) write_op.cc:276] Timestamp: P: 1750859523785560 usec, L: 0
0625 13:52:17.596384 (+ 160us) write_op.cc:270] Start()
0625 13:52:17.596406 (+ 22us) write_op.cc:276] Timestamp: P: 1750859523805076 usec, L: 0
0625 13:52:17.597216 (+ 810us) write_op.cc:270] Start()
0625 13:52:17.597242 (+ 26us) write_op.cc:276] Timestamp: P: 1750859523806920 usec, L: 0
0625 13:52:17.597393 (+ 151us) write_op.cc:270] Start()
0625 13:52:17.597410 (+ 17us) write_op.cc:276] Timestamp: P: 1750859523810093 usec, L: 0
0625 13:52:17.597544 (+ 134us) write_op.cc:270] Start()
0625 13:52:17.597559 (+ 15us) write_op.cc:276] Timestamp: P: 1750859523810653 usec, L: 0
0625 13:52:17.597665 (+ 106us) write_op.cc:270] Start()
0625 13:52:17.597681 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523811753 usec, L: 0
0625 13:52:17.597780 (+ 99us) write_op.cc:270] Start()
0625 13:52:17.597796 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523847861 usec, L: 0
0625 13:52:17.597922 (+ 126us) write_op.cc:270] Start()
0625 13:52:17.597938 (+ 16us) write_op.cc:276] Timestamp: P: 1750859523851412 usec, L: 0
0625 13:52:17.598036 (+ 98us) write_op.cc:270] Start()
0625 13:52:17.598051 (+ 15us) write_op.cc:276] Timestamp: P: 1750859523978078 usec, L: 0
0625 13:52:17.598145 (+ 94us) write_op.cc:270] Start()
0625 13:52:17.598160 (+ 15us) write_op.cc:276] Timestamp: P: 1750859523995006 usec, L: 0
0625 13:52:17.598248 (+ 88us) write_op.cc:270] Start()
0625 13:52:17.598262 (+ 14us) write_op.cc:276] Timestamp: P: 1750859523999068 usec, L: 0
0625 13:52:17.598363 (+ 101us) write_op.cc:270] Start()
0625 13:52:17.598379 (+ 16us) write_op.cc:276] Timestamp: P: 1750859524004401 usec, L: 0
0625 13:52:17.598494 (+ 115us) write_op.cc:270] Start()
0625 13:52:17.598509 (+ 15us) write_op.cc:276] Timestamp: P: 1750859524004971 usec, L: 0
0625 13:52:17.598607 (+ 98us) write_op.cc:270] Start()
0625 13:52:17.598622 (+ 15us) write_op.cc:276] Timestamp: P: 1750859524022447 usec, L: 0
0625 13:52:17.598732 (+ 110us) write_op.cc:270] Start()
0625 13:52:17.598747 (+ 15us) write_op.cc:276] Timestamp: P: 1750859524023393 usec, L: 0
0625 13:52:17.598838 (+ 91us) write_op.cc:270] Start()
0625 13:52:17.598852 (+ 14us) write_op.cc:276] Timestamp: P: 1750859524076750 usec, L: 0
0625 13:52:17.598958 (+ 106us) write_op.cc:270] Start()
0625 13:52:17.598973 (+ 15us) write_op.cc:276] Timestamp: P: 1750859524077824 usec, L: 0
0625 13:52:17.599089 (+ 116us) write_op.cc:270] Start()
0625 13:52:17.599105 (+ 16us) write_op.cc:276] Timestamp: P: 1750859524078617 usec, L: 0
0625 13:52:17.599208 (+ 103us) write_op.cc:270] Start()
0625 13:52:17.599223 (+ 15us) write_op.cc:276] Timestamp: P: 1750859524119066 usec, L: 0
0625 13:52:17.599800 (+ 577us) write_op.cc:270] Start()
0625 13:52:17.599826 (+ 26us) write_op.cc:276] Timestamp: P: 1750859524132077 usec, L: 0
0625 13:52:17.599974 (+ 148us) write_op.cc:270] Start()
0625 13:52:17.599997 (+ 23us) write_op.cc:276] Timestamp: P: 1750859524133338 usec, L: 0
0625 13:52:17.600130 (+ 133us) write_op.cc:270] Start()
0625 13:52:17.600153 (+ 23us) write_op.cc:276] Timestamp: P: 1750859524134146 usec, L: 0
0625 13:52:17.600301 (+ 148us) write_op.cc:270] Start()
0625 13:52:17.600320 (+ 19us) write_op.cc:276] Timestamp: P: 1750859524134955 usec, L: 0
0625 13:52:17.600482 (+ 162us) write_op.cc:270] Start()
0625 13:52:17.600522 (+ 40us) write_op.cc:276] Timestamp: P: 1750859524135718 usec, L: 0
0625 13:52:17.600666 (+ 144us) write_op.cc:270] Start()
0625 13:52:17.600689 (+ 23us) write_op.cc:276] Timestamp: P: 1750859524142611 usec, L: 0
0625 13:52:17.600891 (+ 202us) write_op.cc:270] Start()
0625 13:52:17.600917 (+ 26us) write_op.cc:276] Timestamp: P: 1750859524311900 usec, L: 0
0625 13:52:17.601103 (+ 186us) write_op.cc:270] Start()
0625 13:52:17.601124 (+ 21us) write_op.cc:276] Timestamp: P: 1750859524312421 usec, L: 0
0625 13:52:17.601259 (+ 135us) write_op.cc:270] Start()
0625 13:52:17.601279 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524312848 usec, L: 0
0625 13:52:17.601413 (+ 134us) write_op.cc:270] Start()
0625 13:52:17.601433 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524349164 usec, L: 0
0625 13:52:17.601596 (+ 163us) write_op.cc:270] Start()
0625 13:52:17.601620 (+ 24us) write_op.cc:276] Timestamp: P: 1750859524349776 usec, L: 0
0625 13:52:17.602779 (+ 1159us) write_op.cc:270] Start()
0625 13:52:17.602806 (+ 27us) write_op.cc:276] Timestamp: P: 1750859524350398 usec, L: 0
0625 13:52:17.602934 (+ 128us) write_op.cc:270] Start()
0625 13:52:17.602952 (+ 18us) write_op.cc:276] Timestamp: P: 1750859524369830 usec, L: 0
0625 13:52:17.603068 (+ 116us) write_op.cc:270] Start()
0625 13:52:17.603085 (+ 17us) write_op.cc:276] Timestamp: P: 1750859524377660 usec, L: 0
0625 13:52:17.603213 (+ 128us) write_op.cc:270] Start()
0625 13:52:17.603233 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524378342 usec, L: 0
0625 13:52:17.603384 (+ 151us) write_op.cc:270] Start()
0625 13:52:17.603404 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524380351 usec, L: 0
0625 13:52:17.603550 (+ 146us) write_op.cc:270] Start()
0625 13:52:17.603570 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524397645 usec, L: 0
0625 13:52:17.603691 (+ 121us) write_op.cc:270] Start()
0625 13:52:17.603711 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524398839 usec, L: 0
0625 13:52:17.603855 (+ 144us) write_op.cc:270] Start()
0625 13:52:17.603875 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524456362 usec, L: 0
0625 13:52:17.604008 (+ 133us) write_op.cc:270] Start()
0625 13:52:17.604035 (+ 27us) write_op.cc:276] Timestamp: P: 1750859524460782 usec, L: 0
0625 13:52:17.604198 (+ 163us) write_op.cc:270] Start()
0625 13:52:17.604217 (+ 19us) write_op.cc:276] Timestamp: P: 1750859524509389 usec, L: 0
0625 13:52:17.604343 (+ 126us) write_op.cc:270] Start()
0625 13:52:17.604359 (+ 16us) write_op.cc:276] Timestamp: P: 1750859524513011 usec, L: 0
0625 13:52:17.604504 (+ 145us) write_op.cc:270] Start()
0625 13:52:17.604524 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524522025 usec, L: 0
0625 13:52:17.604635 (+ 111us) write_op.cc:270] Start()
0625 13:52:17.604652 (+ 17us) write_op.cc:276] Timestamp: P: 1750859524560280 usec, L: 0
0625 13:52:17.604778 (+ 126us) write_op.cc:270] Start()
0625 13:52:17.604796 (+ 18us) write_op.cc:276] Timestamp: P: 1750859524568070 usec, L: 0
0625 13:52:17.604919 (+ 123us) write_op.cc:270] Start()
0625 13:52:17.604937 (+ 18us) write_op.cc:276] Timestamp: P: 1750859524568730 usec, L: 0
0625 13:52:17.605065 (+ 128us) write_op.cc:270] Start()
0625 13:52:17.605085 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524606039 usec, L: 0
0625 13:52:17.605229 (+ 144us) write_op.cc:270] Start()
0625 13:52:17.605249 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524606823 usec, L: 0
0625 13:52:17.605387 (+ 138us) write_op.cc:270] Start()
0625 13:52:17.605406 (+ 19us) write_op.cc:276] Timestamp: P: 1750859524627967 usec, L: 0
0625 13:52:17.605556 (+ 150us) write_op.cc:270] Start()
0625 13:52:17.605574 (+ 18us) write_op.cc:276] Timestamp: P: 1750859524667132 usec, L: 0
0625 13:52:17.605692 (+ 118us) write_op.cc:270] Start()
0625 13:52:17.605709 (+ 17us) write_op.cc:276] Timestamp: P: 1750859524714384 usec, L: 0
0625 13:52:17.605820 (+ 111us) write_op.cc:270] Start()
0625 13:52:17.605843 (+ 23us) write_op.cc:276] Timestamp: P: 1750859524725587 usec, L: 0
0625 13:52:17.605963 (+ 120us) write_op.cc:270] Start()
0625 13:52:17.605981 (+ 18us) write_op.cc:276] Timestamp: P: 1750859524830682 usec, L: 0
0625 13:52:17.606124 (+ 143us) write_op.cc:270] Start()
0625 13:52:17.606144 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524873916 usec, L: 0
0625 13:52:17.606268 (+ 124us) write_op.cc:270] Start()
0625 13:52:17.606285 (+ 17us) write_op.cc:276] Timestamp: P: 1750859524876028 usec, L: 0
0625 13:52:17.606409 (+ 124us) write_op.cc:270] Start()
0625 13:52:17.606428 (+ 19us) write_op.cc:276] Timestamp: P: 1750859524876656 usec, L: 0
0625 13:52:17.606548 (+ 120us) write_op.cc:270] Start()
0625 13:52:17.606567 (+ 19us) write_op.cc:276] Timestamp: P: 1750859524877211 usec, L: 0
0625 13:52:17.606680 (+ 113us) write_op.cc:270] Start()
0625 13:52:17.606700 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524877907 usec, L: 0
0625 13:52:17.606842 (+ 142us) write_op.cc:270] Start()
0625 13:52:17.606861 (+ 19us) write_op.cc:276] Timestamp: P: 1750859524879869 usec, L: 0
0625 13:52:17.606997 (+ 136us) write_op.cc:270] Start()
0625 13:52:17.607017 (+ 20us) write_op.cc:276] Timestamp: P: 1750859524880443 usec, L: 0
0625 13:52:17.607181 (+ 164us) write_op.cc:270] Start()
0625 13:52:17.607209 (+ 28us) write_op.cc:276] Timestamp: P: 1750859524880905 usec, L: 0
0625 13:52:17.607397 (+ 188us) write_op.cc:270] Start()
0625 13:52:17.607447 (+ 50us) write_op.cc:276] Timestamp: P: 1750859524881405 usec, L: 0
0625 13:52:17.607602 (+ 155us) write_op.cc:270] Start()
0625 13:52:17.607625 (+ 23us) write_op.cc:276] Timestamp: P: 1750859524881991 usec, L: 0
0625 13:52:17.607769 (+ 144us) write_op.cc:270] Start()
0625 13:52:17.607797 (+ 28us) write_op.cc:276] Timestamp: P: 1750859524905445 usec, L: 0
0625 13:52:17.607912 (+ 115us) write_op.cc:270] Start()
0625 13:52:17.607930 (+ 18us) write_op.cc:276] Timestamp: P: 1750859524972513 usec, L: 0
0625 13:52:17.608080 (+ 150us) write_op.cc:270] Start()
0625 13:52:17.608141 (+ 61us) write_op.cc:276] Timestamp: P: 1750859524973227 usec, L: 0
0625 13:52:17.608306 (+ 165us) write_op.cc:270] Start()
0625 13:52:17.608324 (+ 18us) write_op.cc:276] Timestamp: P: 1750859524973971 usec, L: 0
0625 13:52:17.608436 (+ 112us) write_op.cc:270] Start()
0625 13:52:17.608453 (+ 17us) write_op.cc:276] Timestamp: P: 1750859525067154 usec, L: 0
0625 13:52:17.608626 (+ 173us) write_op.cc:270] Start()
0625 13:52:17.608644 (+ 18us) write_op.cc:276] Timestamp: P: 1750859525067802 usec, L: 0
0625 13:52:17.608767 (+ 123us) write_op.cc:270] Start()
0625 13:52:17.608788 (+ 21us) write_op.cc:276] Timestamp: P: 1750859525068341 usec, L: 0
0625 13:52:17.608963 (+ 175us) write_op.cc:270] Start()
0625 13:52:17.608987 (+ 24us) write_op.cc:276] Timestamp: P: 1750859525069313 usec, L: 0
0625 13:52:17.609151 (+ 164us) write_op.cc:270] Start()
0625 13:52:17.609171 (+ 20us) write_op.cc:276] Timestamp: P: 1750859525070078 usec, L: 0
0625 13:52:17.609304 (+ 133us) write_op.cc:270] Start()
0625 13:52:17.609324 (+ 20us) write_op.cc:276] Timestamp: P: 1750859525101026 usec, L: 0
0625 13:52:17.609455 (+ 131us) write_op.cc:270] Start()
0625 13:52:17.609490 (+ 35us) write_op.cc:276] Timestamp: P: 1750859525195947 usec, L: 0
0625 13:52:17.609649 (+ 159us) write_op.cc:270] Start()
0625 13:52:17.609673 (+ 24us) write_op.cc:276] Timestamp: P: 1750859525197966 usec, L: 0
0625 13:52:17.609821 (+ 148us) write_op.cc:270] Start()
0625 13:52:17.609841 (+ 20us) write_op.cc:276] Timestamp: P: 1750859525211744 usec, L: 0
0625 13:52:17.610018 (+ 177us) write_op.cc:270] Start()
0625 13:52:17.610048 (+ 30us) write_op.cc:276] Timestamp: P: 1750859525219844 usec, L: 0
0625 13:52:17.610172 (+ 124us) write_op.cc:270] Start()
0625 13:52:17.610191 (+ 19us) write_op.cc:276] Timestamp: P: 1750859525220740 usec, L: 0
0625 13:52:17.610661 (+ 470us) write_op.cc:270] Start()
0625 13:52:17.610690 (+ 29us) write_op.cc:276] Timestamp: P: 1750859525221569 usec, L: 0
0625 13:52:17.610842 (+ 152us) write_op.cc:270] Start()
0625 13:52:17.610865 (+ 23us) write_op.cc:276] Timestamp: P: 1750859525222511 usec, L: 0
0625 13:52:17.611006 (+ 141us) write_op.cc:270] Start()
0625 13:52:17.611322 (+ 316us) write_op.cc:276] Timestamp: P: 1750859525223259 usec, L: 0
0625 13:52:17.611536 (+ 214us) write_op.cc:270] Start()
0625 13:52:17.611561 (+ 25us) write_op.cc:276] Timestamp: P: 1750859525308311 usec, L: 0
0625 13:52:17.611699 (+ 138us) write_op.cc:270] Start()
0625 13:52:17.611717 (+ 18us) write_op.cc:276] Timestamp: P: 1750859525332840 usec, L: 0
0625 13:52:17.611823 (+ 106us) write_op.cc:270] Start()
0625 13:52:17.611840 (+ 17us) write_op.cc:276] Timestamp: P: 1750859525345208 usec, L: 0
0625 13:52:17.612350 (+ 510us) write_op.cc:270] Start()
0625 13:52:17.612378 (+ 28us) write_op.cc:276] Timestamp: P: 1750859525360179 usec, L: 0
0625 13:52:17.612570 (+ 192us) write_op.cc:270] Start()
0625 13:52:17.612591 (+ 21us) write_op.cc:276] Timestamp: P: 1750859525381115 usec, L: 0
0625 13:52:17.612721 (+ 130us) write_op.cc:270] Start()
0625 13:52:17.612741 (+ 20us) write_op.cc:276] Timestamp: P: 1750859525397316 usec, L: 0
0625 13:52:17.612871 (+ 130us) write_op.cc:270] Start()
0625 13:52:17.612890 (+ 19us) write_op.cc:276] Timestamp: P: 1750859525519770 usec, L: 0
0625 13:52:17.613026 (+ 136us) write_op.cc:270] Start()
0625 13:52:17.613045 (+ 19us) write_op.cc:276] Timestamp: P: 1750859525520454 usec, L: 0
0625 13:52:17.613182 (+ 137us) write_op.cc:270] Start()
0625 13:52:17.613201 (+ 19us) write_op.cc:276] Timestamp: P: 1750859525521014 usec, L: 0
0625 13:52:17.613375 (+ 174us) write_op.cc:270] Start()
0625 13:52:17.613396 (+ 21us) write_op.cc:276] Timestamp: P: 1750859525521551 usec, L: 0
0625 13:52:17.613535 (+ 139us) write_op.cc:270] Start()
0625 13:52:17.613554 (+ 19us) write_op.cc:276] Timestamp: P: 1750859525522135 usec, L: 0
0625 13:52:17.613695 (+ 141us) write_op.cc:270] Start()
0625 13:52:17.613714 (+ 19u
I20250625 13:52:17.931517 1165 tablet_replica.cc:331] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:17.931820 1165 raft_consensus.cc:2241] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.933033 1165 raft_consensus.cc:2270] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.933506 1170 tablet_bootstrap.cc:492] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:17.933812 1170 ts_tablet_manager.cc:1397] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.003s
I20250625 13:52:17.934645 1166 tablet_bootstrap.cc:492] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:17.934845 1166 ts_tablet_manager.cc:1397] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250625 13:52:17.934899 1170 raft_consensus.cc:357] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.935230 1170 raft_consensus.cc:383] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.935380 1170 raft_consensus.cc:738] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.935588 1166 raft_consensus.cc:357] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.935716 1170 consensus_queue.cc:260] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.935829 1166 raft_consensus.cc:383] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:17.935890 1166 raft_consensus.cc:738] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:17.936126 1166 consensus_queue.cc:260] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:17.937357 1166 ts_tablet_manager.cc:1428] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:17.938400 1168 ts_tablet_manager.cc:1918] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250625 13:52:17.939543 1166 tablet_bootstrap.cc:492] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:17.940917 1166 tablet_bootstrap.cc:654] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:17.941042 1165 ts_tablet_manager.cc:1905] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.948355 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 8063122ea7b24b6790f5c3cc884e09be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:17.948509 1165 ts_tablet_manager.cc:1918] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:17.950132 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ef2b08fba37f4a38880c018ec2d869c8 (table default.Table_3a4e5f10c2ec453a82fb97800a0d37f7 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]). Expected version 2 got 0
I20250625 13:52:17.956295 593 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
I20250625 13:52:17.958777 593 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 5a289b9e3fac4fd8bacc22c235f7010c (table default.Table_1fafa11a1445445081345630d89f80dc [id=8e9a2d87890f4b5985c342a76d21eb5a]). Expected version 1 got 0
I20250625 13:52:17.965181 1165 log.cc:1199] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/afa1c198dd6541809843244c7cc18194
I20250625 13:52:17.965821 1165 ts_tablet_manager.cc:1939] T afa1c198dd6541809843244c7cc18194 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:17.971850 1165 tablet_replica.cc:331] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:17.972540 1165 raft_consensus.cc:2241] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:17.972956 1165 raft_consensus.cc:2270] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:17.973884 1165 ts_tablet_manager.cc:1905] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:17.964797 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8e4df287c95946e2beb94524073e5ee2
I20250625 13:52:17.975934 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3a318be2bb4949268060c27e670001ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.976507 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a318be2bb4949268060c27e670001ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.964993 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 663588b7e9ee430faa180fec00897e22
I20250625 13:52:17.978505 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9998378b240c46bcb121279c983960cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.978972 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9998378b240c46bcb121279c983960cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:17.965243 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05f205733f3d4c81b9e71709c7559660 consensus state has no leader
I20250625 13:52:17.967276 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet afa1c198dd6541809843244c7cc18194 (table default.Table_c0ca5c113cfd4c3794ceb90cdc88bb94 [id=3c2aabfc6e104537ac703fbcf59f2470]) successfully deleted
I20250625 13:52:17.980358 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_56e5c59a581d431cada560e8606965ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:17.980796 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56e5c59a581d431cada560e8606965ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:17.980824 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e4df287c95946e2beb94524073e5ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.980964 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 663588b7e9ee430faa180fec00897e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:17.983844 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95dfea44fd4a4591b9363670b2ababe9 consensus state has no leader
I20250625 13:52:17.984052 588 master_service.cc:948] ReplaceTablet: received request to replace tablet c431abffd05c4136b7510c60eee2a8b1 from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:17.984532 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a079e985358040e78455ac8cb6e49f16 consensus state has no leader
I20250625 13:52:17.987571 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 669de0b7c93a4efea8623dab2e462c8d (table default.Table_8315e81e0d0843819735795f79fa91fb [id=d3092ac7d43a4e9891fe289b304e937c]). Expected version 3 got 0
I20250625 13:52:17.994354 593 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 5a289b9e3fac4fd8bacc22c235f7010c (table default.Table_1fafa11a1445445081345630d89f80dc [id=8e9a2d87890f4b5985c342a76d21eb5a]). Expected version 1 got 0
I20250625 13:52:17.996513 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 7efc8df3c8e24020b073a1b61bc9e0a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f06030f45284c5a9929a4c4513004ba) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:17.941025 1171 tablet_bootstrap.cc:492] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:17.976279 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e4df287c95946e2beb94524073e5ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.978734 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 663588b7e9ee430faa180fec00897e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:17.988088 2204 consensus_queue.cc:1035] T dec08d20a09444f5b9a102a8712e36c7 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:17.998674 1171 tablet_bootstrap.cc:654] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.000029 2204 raft_consensus.cc:491] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.000172 2204 raft_consensus.cc:513] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.000751 2204 leader_election.cc:290] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.001791 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2b08fba37f4a38880c018ec2d869c8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:18.001976 778 raft_consensus.cc:2466] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:18.002326 851 leader_election.cc:304] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.002771 2204 raft_consensus.cc:2802] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.002841 2204 raft_consensus.cc:491] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.002903 2204 raft_consensus.cc:3058] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.004082 1168 log.cc:1199] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/3f56a8a6cb964d3a979e2f1d944eeddc
I20250625 13:52:18.004452 1168 ts_tablet_manager.cc:1939] T 3f56a8a6cb964d3a979e2f1d944eeddc P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:18.005192 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1198) took 3446 ms. Trace:
I20250625 13:52:18.005265 1168 rpcz_store.cc:276] 0625 13:52:14.559173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.581483 (+ 22310us) service_pool.cc:225] Handling call
0625 13:52:17.891155 (+3309672us) ts_tablet_manager.cc:1145] Deleting tablet 3f56a8a6cb964d3a979e2f1d944eeddc
0625 13:52:17.899299 (+ 8144us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.935240 (+ 35941us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:17.938307 (+ 3067us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.005174 (+ 66867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309497}
I20250625 13:52:18.005681 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 3f56a8a6cb964d3a979e2f1d944eeddc (table default.Table_2c311e79aef2487d8524720a4cd0677c [id=71729b0c8f7046bb8e80bb96f9f2ba71]) successfully deleted
I20250625 13:52:18.005741 2204 raft_consensus.cc:513] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.006247 2471 raft_consensus.cc:491] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.006672 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet a29708fb688a4a65b84ae2564a040b4c with delete_type TABLET_DATA_DELETED (replaced by tablet 2c8cf5d6cd624a1797486c69bd2098fd) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.006839 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e4df287c95946e2beb94524073e5ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.006938 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 663588b7e9ee430faa180fec00897e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:17 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.007035 2556 raft_consensus.cc:491] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.007150 1170 ts_tablet_manager.cc:1428] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.073s user 0.001s sys 0.001s
I20250625 13:52:18.007192 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2b08fba37f4a38880c018ec2d869c8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.006420 2471 raft_consensus.cc:513] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.007588 1170 tablet_bootstrap.cc:492] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:18.008042 2471 leader_election.cc:290] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:18.009552 849 leader_election.cc:343] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.007165 2556 raft_consensus.cc:513] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.016481 1165 ts_tablet_manager.cc:1918] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.016697 1165 log.cc:1199] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/0e3ef7f5b474466bbc19fd31c6b6d99c
I20250625 13:52:18.017238 2204 leader_election.cc:290] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.017581 1166 tablet_bootstrap.cc:492] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:18.017699 1166 ts_tablet_manager.cc:1397] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.007s
I20250625 13:52:18.018496 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2b08fba37f4a38880c018ec2d869c8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:18.018658 778 raft_consensus.cc:3058] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.019097 1166 raft_consensus.cc:357] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.019237 1166 raft_consensus.cc:383] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.019279 1166 raft_consensus.cc:738] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.019475 1166 consensus_queue.cc:260] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.019912 1166 ts_tablet_manager.cc:1428] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:18.020699 1166 tablet_bootstrap.cc:492] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:18.021320 778 raft_consensus.cc:2466] T ef2b08fba37f4a38880c018ec2d869c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:18.022046 851 leader_election.cc:304] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.024482 2518 raft_consensus.cc:2802] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:17.984702 2523 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d85834f366fa42c294c9290af21ea111)
I20250625 13:52:18.027598 2523 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:18.029825 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 553af7c7326240838f37e3861d4e3357 with delete_type TABLET_DATA_DELETED (replaced by tablet bf7463962f39468c85fb3dc2edcf608f) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.030220 2523 leader_election.cc:290] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58351267767f4d9a85f974ebc40db6fe (127.0.104.61:45727), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
W20250625 13:52:18.033521 985 leader_election.cc:343] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8063122ea7b24b6790f5c3cc884e09be
W20250625 13:52:18.033927 983 leader_election.cc:343] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 8063122ea7b24b6790f5c3cc884e09be
I20250625 13:52:18.035001 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2b08fba37f4a38880c018ec2d869c8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:18.035630 849 leader_election.cc:343] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.037323 983 leader_election.cc:304] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.037606 2471 raft_consensus.cc:2747] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.038120 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe4046742b34f1ab0b1610b5d6ee01e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.038306 913 raft_consensus.cc:2466] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:18.038427 2254 ts_tablet_manager.cc:1918] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:18.037874 2556 leader_election.cc:290] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.038905 983 leader_election.cc:304] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.039441 2556 raft_consensus.cc:2802] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.041096 1170 tablet_bootstrap.cc:654] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.043200 2068 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" instance_seqno: 1750859510146049) as {username='slave'} at 127.0.104.1:42367; Asking this server to re-register.
I20250625 13:52:18.043490 2254 log.cc:1199] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/60ef50a3624b4c79acc803ff0e2ed364
I20250625 13:52:18.044458 2556 raft_consensus.cc:491] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.044898 2556 raft_consensus.cc:3058] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.048063 2556 raft_consensus.cc:513] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.049554 2556 leader_election.cc:290] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.050093 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe4046742b34f1ab0b1610b5d6ee01e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:18.050333 913 raft_consensus.cc:3058] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:18.050576 985 leader_election.cc:343] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cbe4046742b34f1ab0b1610b5d6ee01e
W20250625 13:52:18.051429 985 leader_election.cc:343] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: cbe4046742b34f1ab0b1610b5d6ee01e
I20250625 13:52:18.051793 2254 ts_tablet_manager.cc:1939] T 60ef50a3624b4c79acc803ff0e2ed364 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:18.052800 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1281) took 1386 ms. Trace:
I20250625 13:52:18.052904 2254 rpcz_store.cc:276] 0625 13:52:16.665850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.665902 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:17.833773 (+1167871us) ts_tablet_manager.cc:1145] Deleting tablet 60ef50a3624b4c79acc803ff0e2ed364
0625 13:52:17.834334 (+ 561us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:17.949733 (+115399us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.038287 (+ 88554us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.052758 (+ 14471us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167059}
I20250625 13:52:18.053154 913 raft_consensus.cc:2466] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:18.053608 983 leader_election.cc:304] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.054185 2556 raft_consensus.cc:2802] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:18.054457 2556 raft_consensus.cc:695] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:18.054769 2556 consensus_queue.cc:237] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.055657 2254 tablet_replica.cc:331] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:18.055948 2254 raft_consensus.cc:2241] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:18.056221 2254 pending_rounds.cc:70] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:18.056308 2254 pending_rounds.cc:77] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520666784415744 op_type: NO_OP noop_request { }
I20250625 13:52:18.056434 2254 raft_consensus.cc:2887] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:18.059798 2254 raft_consensus.cc:2270] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.061235 823 heartbeater.cc:461] Registering TS with master...
I20250625 13:52:18.061784 2254 ts_tablet_manager.cc:1905] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.067552 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0221d3b2f7554e1a9a8159c7fade3b33" } new_table_name: "default.Table_3394a083acc4469a824af3cb9f150ba7" modify_external_catalogs: true
I20250625 13:52:18.069213 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_deb3b74de40448608a132c85f19617d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.069711 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_deb3b74de40448608a132c85f19617d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.070137 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_51a539bc437b40f6ae158307f8306a0f" } modify_external_catalogs: true
I20250625 13:52:18.070302 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_51a539bc437b40f6ae158307f8306a0f" } modify_external_catalogs: true
I20250625 13:52:18.070792 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a18783dacb0b4bd09053d2559ed93252" } modify_external_catalogs: true
I20250625 13:52:18.070917 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a18783dacb0b4bd09053d2559ed93252" } modify_external_catalogs: true
I20250625 13:52:18.072992 1166 tablet_bootstrap.cc:654] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.074761 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_938b272613694e4bb9b163d19e37053a" } new_table_name: "default.Table_c6c55b9a2baa4454b3ea4e9beb218714" modify_external_catalogs: true
I20250625 13:52:18.076462 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2600471f74cf412b96ad3545d0f94cba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.077051 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2600471f74cf412b96ad3545d0f94cba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.080627 1165 ts_tablet_manager.cc:1939] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.081274 2548 raft_consensus.cc:491] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.081420 2548 raft_consensus.cc:513] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.082144 2548 leader_election.cc:290] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.082850 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a289b9e3fac4fd8bacc22c235f7010c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.083490 711 leader_election.cc:343] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.084254 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a289b9e3fac4fd8bacc22c235f7010c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:18.084833 711 leader_election.cc:343] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.084946 711 leader_election.cc:304] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.085618 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff276efb7beb42dcaf8cb45ca0c0e04a" } modify_external_catalogs: true
I20250625 13:52:18.085845 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ff276efb7beb42dcaf8cb45ca0c0e04a" } modify_external_catalogs: true
I20250625 13:52:18.087631 2548 raft_consensus.cc:2747] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.089339 2068 ts_manager.cc:194] Registered new tserver with Master: 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.092113 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1303) took 1028 ms. Trace:
I20250625 13:52:18.092228 1165 rpcz_store.cc:276] 0625 13:52:17.064014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.064061 (+ 47us) service_pool.cc:225] Handling call
0625 13:52:17.971792 (+907731us) ts_tablet_manager.cc:1145] Deleting tablet 0e3ef7f5b474466bbc19fd31c6b6d99c
0625 13:52:17.971803 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.001503 (+ 29700us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.010662 (+ 9159us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.092090 (+ 81428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":907580}
I20250625 13:52:18.092626 1165 tablet_replica.cc:331] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.092752 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 0e3ef7f5b474466bbc19fd31c6b6d99c (table default.Table_30da1f016b424e85bcbae236fafd8060 [id=1f2d72b34f7a4cddb7555f2ddc708a42]) successfully deleted
I20250625 13:52:18.092823 1165 raft_consensus.cc:2241] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.092981 1165 raft_consensus.cc:2270] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.093541 1165 ts_tablet_manager.cc:1905] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.097793 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1199) took 3488 ms. Trace:
I20250625 13:52:18.097978 1168 rpcz_store.cc:276] 0625 13:52:14.609411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.609472 (+ 61us) service_pool.cc:225] Handling call
0625 13:52:18.097604 (+3488132us) ts_tablet_manager.cc:1145] Deleting tablet bb9fcf91fa304a359213a3e5267a9cac
0625 13:52:18.097620 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.097763 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3487917}
I20250625 13:52:18.098559 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1200) took 3486 ms. Trace:
I20250625 13:52:18.098713 1168 rpcz_store.cc:276] 0625 13:52:14.612238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.613600 (+ 1362us) service_pool.cc:225] Handling call
0625 13:52:18.098443 (+3484843us) ts_tablet_manager.cc:1145] Deleting tablet 83699efa521a44f092cb738002aa58e4
0625 13:52:18.098453 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.098542 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3484597}
I20250625 13:52:18.099211 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1201) took 3485 ms. Trace:
I20250625 13:52:18.099354 1168 rpcz_store.cc:276] 0625 13:52:14.613762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.613878 (+ 116us) service_pool.cc:225] Handling call
0625 13:52:18.099112 (+3485234us) ts_tablet_manager.cc:1145] Deleting tablet 0346ed4f8d3f46aa84cbabbcadcebf23
0625 13:52:18.099120 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.099195 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3485087}
I20250625 13:52:18.099925 1168 tablet_replica.cc:331] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:18.100219 1168 raft_consensus.cc:2241] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.100479 1168 raft_consensus.cc:2270] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.101087 1168 ts_tablet_manager.cc:1905] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:18.107846 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 149481us
W20250625 13:52:18.108969 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 62951.7us
I20250625 13:52:18.118873 2068 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" instance_seqno: 1750859511100740) as {username='slave'} at 127.0.104.3:35145; Asking this server to re-register.
W20250625 13:52:18.119912 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 589a42f0f7b94ae88d09d0f7de60e175 consensus state has no leader
I20250625 13:52:18.121848 1095 heartbeater.cc:461] Registering TS with master...
I20250625 13:52:18.127835 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 39540b81fd4746858e65be49ded0e1df from {username='slave'} at 127.0.0.1:54762
W20250625 13:52:18.130553 849 consensus_peers.cc:487] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:18.131764 1171 tablet_bootstrap.cc:492] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.131994 1171 ts_tablet_manager.cc:1397] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.191s user 0.002s sys 0.009s
I20250625 13:52:18.132905 1171 raft_consensus.cc:357] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.133213 1171 raft_consensus.cc:383] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.133222 2068 ts_manager.cc:194] Registered new tserver with Master: 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:18.133350 1171 raft_consensus.cc:738] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.133719 1171 consensus_queue.cc:260] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.134752 1171 ts_tablet_manager.cc:1428] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:18.135205 1171 tablet_bootstrap.cc:492] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:18.136045 1166 tablet_bootstrap.cc:492] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:18.136242 1166 ts_tablet_manager.cc:1397] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.003s
I20250625 13:52:18.136936 2518 raft_consensus.cc:695] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:18.137207 1166 raft_consensus.cc:357] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.137375 1166 raft_consensus.cc:383] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.137434 1166 raft_consensus.cc:738] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.137259 2518 consensus_queue.cc:237] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.137657 1166 consensus_queue.cc:260] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.138438 1166 ts_tablet_manager.cc:1428] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250625 13:52:18.139815 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bb9fcf91fa304a359213a3e5267a9cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb9fcf91fa304a359213a3e5267a9cac already in progress: creating tablet
W20250625 13:52:18.140019 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 83699efa521a44f092cb738002aa58e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83699efa521a44f092cb738002aa58e4 already in progress: creating tablet
W20250625 13:52:18.140187 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 0346ed4f8d3f46aa84cbabbcadcebf23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0346ed4f8d3f46aa84cbabbcadcebf23 already in progress: creating tablet
I20250625 13:52:18.140478 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 60ef50a3624b4c79acc803ff0e2ed364 (table default.Table_3e7baa9cc87143b0833a6bc32a193408 [id=d043c74a6cb94efa89ce819d2ef35545]) successfully deleted
I20250625 13:52:18.146759 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 9be75d03121f4ea2906236ac3a5b0764 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.150027 1170 tablet_bootstrap.cc:492] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:18.150193 1170 ts_tablet_manager.cc:1397] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.143s user 0.001s sys 0.010s
I20250625 13:52:18.150892 1170 raft_consensus.cc:357] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.151050 1170 raft_consensus.cc:383] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.151124 1170 raft_consensus.cc:738] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.151419 1170 consensus_queue.cc:260] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.152042 1170 ts_tablet_manager.cc:1428] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:18.152531 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d336a3fe94264da6994165e69d9daa99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.153209 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d336a3fe94264da6994165e69d9daa99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.154239 1170 tablet_bootstrap.cc:492] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:18.155167 2254 ts_tablet_manager.cc:1918] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:18.159466 2254 log.cc:1199] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/dcf7fab53cae49908d2952cdaa07943a
I20250625 13:52:18.160094 2254 ts_tablet_manager.cc:1939] T dcf7fab53cae49908d2952cdaa07943a P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:18.170678 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1282) took 1429 ms. Trace:
I20250625 13:52:18.157802 1170 tablet_bootstrap.cc:654] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.173918 2254 rpcz_store.cc:276] 0625 13:52:16.741376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.741430 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:18.055582 (+1314152us) ts_tablet_manager.cc:1145] Deleting tablet dcf7fab53cae49908d2952cdaa07943a
0625 13:52:18.055595 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.098818 (+ 43223us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.154970 (+ 56152us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.170622 (+ 15652us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1313960}
I20250625 13:52:18.181690 2254 tablet_replica.cc:331] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:18.182109 2254 raft_consensus.cc:2241] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:18.182489 2254 pending_rounds.cc:70] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:18.182687 2254 pending_rounds.cc:77] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520664777920512 op_type: NO_OP noop_request { }
I20250625 13:52:18.182897 2254 raft_consensus.cc:2887] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:18.183082 2254 raft_consensus.cc:2270] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.189713 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet dcf7fab53cae49908d2952cdaa07943a (table default.Table_fe872dae10544c0ca6a6735266cd03d2 [id=aa4ef3fe7bbb408396878d79f7caffb9]) successfully deleted
I20250625 13:52:18.191936 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c52f47756979445e9c05f1f68d2283cd" } modify_external_catalogs: true
I20250625 13:52:18.192010 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ec46a4ca51884b13aa80806684369744"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:18.192169 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c52f47756979445e9c05f1f68d2283cd" } modify_external_catalogs: true
W20250625 13:52:18.192497 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec46a4ca51884b13aa80806684369744 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.194048 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fdada5e5d21042ecb3d4b84dc1f0b299" } modify_external_catalogs: true
I20250625 13:52:18.194232 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_fdada5e5d21042ecb3d4b84dc1f0b299" } modify_external_catalogs: true
I20250625 13:52:18.195254 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1121 } } ignore_live_leader: false dest_uuid: "58351267767f4d9a85f974ebc40db6fe" is_pre_election: true
I20250625 13:52:18.197114 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 63d01af6b320489fb99c467c11ed1dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.197660 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_8be17994b5324638bcceef7134adbaf1" } new_table_name: "default.Table_869cfe1552a84a369de360408df7f656" modify_external_catalogs: true
I20250625 13:52:18.203899 1165 ts_tablet_manager.cc:1918] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.204156 1165 log.cc:1199] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/ef2c3757c0284aaca70ec317c05195d6
I20250625 13:52:18.204617 1165 ts_tablet_manager.cc:1939] T ef2c3757c0284aaca70ec317c05195d6 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.206218 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1304) took 1141 ms. Trace:
I20250625 13:52:18.206351 1165 rpcz_store.cc:276] 0625 13:52:17.064558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.064638 (+ 80us) service_pool.cc:225] Handling call
0625 13:52:18.092570 (+1027932us) ts_tablet_manager.cc:1145] Deleting tablet ef2c3757c0284aaca70ec317c05195d6
0625 13:52:18.092583 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.155736 (+ 63153us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.203763 (+ 48027us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.206197 (+ 2434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027774}
I20250625 13:52:18.206943 1165 tablet_replica.cc:331] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.207154 1165 raft_consensus.cc:2241] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.207340 1165 raft_consensus.cc:2270] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.207875 1165 ts_tablet_manager.cc:1905] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.208427 1166 tablet_bootstrap.cc:492] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:18.206820 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet ef2c3757c0284aaca70ec317c05195d6 (table default.Table_44a75b53cab0470da32c4aedecdc2434 [id=7e8ccd792ea1452ba8c9fb719d11de52]) successfully deleted
I20250625 13:52:18.198740 2425 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d85834f366fa42c294c9290af21ea111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:18.210583 1166 tablet_bootstrap.cc:654] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.210711 2425 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111 [sys.catalog]: This master's current role is: LEADER
I20250625 13:52:18.210803 2254 ts_tablet_manager.cc:1905] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.224640 2068 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "affa1c3035b940e88ca143be7d35840d" instance_seqno: 1750859510608079) as {username='slave'} at 127.0.104.2:57913; Asking this server to re-register.
I20250625 13:52:18.226756 961 heartbeater.cc:461] Registering TS with master...
I20250625 13:52:18.228123 1170 tablet_bootstrap.cc:492] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:18.228299 1170 ts_tablet_manager.cc:1397] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20250625 13:52:18.229138 1170 raft_consensus.cc:357] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.229313 1170 raft_consensus.cc:383] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.229382 1170 raft_consensus.cc:738] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.229609 1170 consensus_queue.cc:260] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.230152 1170 ts_tablet_manager.cc:1428] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:18.233772 1170 tablet_bootstrap.cc:492] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:18.235823 1170 tablet_bootstrap.cc:654] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.238417 2548 raft_consensus.cc:491] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.238564 2548 raft_consensus.cc:513] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.238907 2254 ts_tablet_manager.cc:1918] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:18.239049 2254 log.cc:1199] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/3859f399a7b84de99658380bd6c24ff5
I20250625 13:52:18.239216 2548 leader_election.cc:290] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.239457 2254 ts_tablet_manager.cc:1939] T 3859f399a7b84de99658380bd6c24ff5 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:18.239924 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed3c177bb6dd488aa54bee2a447016ff" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.240018 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1121 } } ignore_live_leader: false dest_uuid: "d85834f366fa42c294c9290af21ea111" is_pre_election: true
I20250625 13:52:18.240530 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed3c177bb6dd488aa54bee2a447016ff" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.243453 711 leader_election.cc:343] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:18.243652 711 leader_election.cc:343] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.243748 711 leader_election.cc:304] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.244683 2548 raft_consensus.cc:2747] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.248703 1999 leader_election.cc:304] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fec9d8a79054a5991ae90d8a3101e32; no voters: 58351267767f4d9a85f974ebc40db6fe, d85834f366fa42c294c9290af21ea111
I20250625 13:52:18.250020 1171 tablet_bootstrap.cc:654] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.261245 1168 ts_tablet_manager.cc:1918] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.261293 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1283) took 1452 ms. Trace:
I20250625 13:52:18.261525 1168 log.cc:1199] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/60ef50a3624b4c79acc803ff0e2ed364
I20250625 13:52:18.261404 2254 rpcz_store.cc:276] 0625 13:52:16.808444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.808496 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:18.181591 (+1373095us) ts_tablet_manager.cc:1145] Deleting tablet 3859f399a7b84de99658380bd6c24ff5
0625 13:52:18.181606 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.214070 (+ 32464us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.238777 (+ 24707us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.261244 (+ 22467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372922}
I20250625 13:52:18.261794 2254 tablet_replica.cc:331] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:18.261922 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 3859f399a7b84de99658380bd6c24ff5 (table default.Table_cf29f4b4bbb94091affd4440a4b8447c [id=27acc786d57a4c7ab531242dd52f17ab]) successfully deleted
I20250625 13:52:18.262117 2254 raft_consensus.cc:2241] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.262364 2254 raft_consensus.cc:2270] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.263109 2254 ts_tablet_manager.cc:1905] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.265049 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d92fe17f576e491eb189a10ca72a7e09
I20250625 13:52:18.265484 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 6c89a899837f40a8ac9eed9f1d68e420
I20250625 13:52:18.265779 757 tablet_service.cc:1515] Processing DeleteTablet for tablet d92fe17f576e491eb189a10ca72a7e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.266433 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c89a899837f40a8ac9eed9f1d68e420 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:18.267184 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f06030f45284c5a9929a4c4513004ba consensus state has no leader
I20250625 13:52:18.269924 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d92fe17f576e491eb189a10ca72a7e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.270146 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8a9b6e6e1f2d41288a958115dc18d4fb
I20250625 13:52:18.270787 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a9b6e6e1f2d41288a958115dc18d4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.271087 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c89a899837f40a8ac9eed9f1d68e420 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.271471 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b29ecaa200d14dc583380e80f8bae166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.271996 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b29ecaa200d14dc583380e80f8bae166 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.273243 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7ba3974781cc491bb035136be8634179"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.273691 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ba3974781cc491bb035136be8634179 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.273722 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 8dd0d216fb394f31afa4478ad2097fe5
I20250625 13:52:18.274441 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd0d216fb394f31afa4478ad2097fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:18.274564 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ecd109dacfcb48a7a5ab54ef6f85a1b1 consensus state has no leader
I20250625 13:52:18.275082 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3e78c988506748b8b316cc58f5c6a9dc" } new_table_name: "default.Table_3abea715292340d09a0301ff4832bbdc" modify_external_catalogs: true
I20250625 13:52:18.275593 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_18f5eaac4ead4aa8bb9e663f4fec5d79" } new_table_name: "default.Table_9d292e9cae0d40fe8cb032e80b18b010" modify_external_catalogs: true
I20250625 13:52:18.277216 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3348879866c946209d1b8e1d87153e2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.277732 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3348879866c946209d1b8e1d87153e2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.280735 2556 raft_consensus.cc:491] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.280894 2556 raft_consensus.cc:513] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.281536 2556 leader_election.cc:290] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:18.282832 985 leader_election.cc:343] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 70959ec681524518a7ec2c9ed4016ad1
I20250625 13:52:18.284431 596 catalog_manager.cc:5582] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:18.284879 596 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ef2b08fba37f4a38880c018ec2d869c8 (table default.Table_3a4e5f10c2ec453a82fb97800a0d37f7 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]). Expected version 2 got 0
I20250625 13:52:18.287503 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a9b6e6e1f2d41288a958115dc18d4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.288548 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd0d216fb394f31afa4478ad2097fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.290247 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a30a35e3639a43c5b078169060824723 with delete_type TABLET_DATA_DELETED (replaced by tablet efc9536a716f4b9f9fc4af1637b3cd80) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:18.294065 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 564f4e65472c4e499f26bb813029db22 consensus state has no leader
W20250625 13:52:18.295014 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9700337fd5984819a8b2f16b28c2ccac consensus state has no leader
I20250625 13:52:18.270537 889 tablet_service.cc:1515] Processing DeleteTablet for tablet d92fe17f576e491eb189a10ca72a7e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.273258 2518 consensus_queue.cc:1035] T 647c97a8ae484c4a8694e777bc62d602 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:18.309275 1168 ts_tablet_manager.cc:1939] T 60ef50a3624b4c79acc803ff0e2ed364 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:18.282527 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70959ec681524518a7ec2c9ed4016ad1" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.309542 913 raft_consensus.cc:2466] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:18.286837 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c89a899837f40a8ac9eed9f1d68e420 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.288158 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a9b6e6e1f2d41288a958115dc18d4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.289147 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd0d216fb394f31afa4478ad2097fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.302479 2573 consensus_queue.cc:1035] T 38770320fc6541c39e10448978f8a3b0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:18.310382 983 leader_election.cc:304] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.311129 2556 raft_consensus.cc:2802] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.311256 2556 raft_consensus.cc:491] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.311388 2556 raft_consensus.cc:3058] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.312537 2569 catalog_manager.cc:1261] Loaded cluster ID: 4fb72e0fa75741a1972554db6ec14e0e
I20250625 13:52:18.312618 2569 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: loading cluster ID for follower catalog manager: success
I20250625 13:52:18.314966 2556 raft_consensus.cc:513] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.314905 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_01a81763e3d0469c86e102451a0e9151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.315510 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01a81763e3d0469c86e102451a0e9151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.322621 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5b7b7a3dd6cf4f0a9cc2be1c0664cc96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.323169 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b7b7a3dd6cf4f0a9cc2be1c0664cc96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.324389 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_25bef492a0bc4aafa4f94e4917f3ab79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.324913 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25bef492a0bc4aafa4f94e4917f3ab79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.326920 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9150143ab7394f1baf2fb087e8265dfa" } new_table_name: "default.Table_40d641f883394a99942efff5d105c72a" modify_external_catalogs: true
I20250625 13:52:18.328197 1166 tablet_bootstrap.cc:492] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:18.328429 1166 ts_tablet_manager.cc:1397] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.003s
I20250625 13:52:18.329350 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b64d5e1e2ea54fbdbf2fd4d9cdea9fe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:18.329435 1166 raft_consensus.cc:357] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.329644 1166 raft_consensus.cc:383] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.329777 1166 raft_consensus.cc:738] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:18.329993 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b64d5e1e2ea54fbdbf2fd4d9cdea9fe4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.330183 1166 consensus_queue.cc:260] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.331023 1166 ts_tablet_manager.cc:1428] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:18.331444 1166 tablet_bootstrap.cc:492] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:18.333071 985 leader_election.cc:343] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 70959ec681524518a7ec2c9ed4016ad1
I20250625 13:52:18.333027 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70959ec681524518a7ec2c9ed4016ad1" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:18.333241 913 raft_consensus.cc:3058] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.334733 1166 tablet_bootstrap.cc:654] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.335582 2556 leader_election.cc:290] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.336680 913 raft_consensus.cc:2466] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:18.337091 983 leader_election.cc:304] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.337174 1171 tablet_bootstrap.cc:492] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.337392 1171 ts_tablet_manager.cc:1397] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.202s user 0.004s sys 0.002s
I20250625 13:52:18.338311 1171 raft_consensus.cc:357] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:18.338598 1171 raft_consensus.cc:383] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.338703 1171 raft_consensus.cc:738] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.338819 2556 raft_consensus.cc:2802] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:18.338948 2556 raft_consensus.cc:695] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:18.339067 1171 consensus_queue.cc:260] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:18.339196 2556 consensus_queue.cc:237] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.344430 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 2191b836bb3044c89fee13c869d9e07d from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:18.349053 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249c5c2aa564027a2c594ca4c31e1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.349301 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 829221b8a79f460d90dd24be6a2d4c43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.356036 2548 raft_consensus.cc:491] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.356204 2548 raft_consensus.cc:513] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.356781 2548 leader_election.cc:290] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.357923 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71cf3b95139245a4ba11b6bcaa3ab3f4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.358012 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71cf3b95139245a4ba11b6bcaa3ab3f4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.358364 711 leader_election.cc:343] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:18.358547 711 leader_election.cc:343] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.358618 711 leader_election.cc:304] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.359038 2548 raft_consensus.cc:2747] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.367470 1165 ts_tablet_manager.cc:1918] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.367700 1165 log.cc:1199] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/3249c5c2aa564027a2c594ca4c31e1c8
I20250625 13:52:18.368185 1165 ts_tablet_manager.cc:1939] T 3249c5c2aa564027a2c594ca4c31e1c8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.369115 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1305) took 1303 ms. Trace:
I20250625 13:52:18.369262 1165 rpcz_store.cc:276] 0625 13:52:17.065424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.065508 (+ 84us) service_pool.cc:225] Handling call
0625 13:52:18.206898 (+1141390us) ts_tablet_manager.cc:1145] Deleting tablet 3249c5c2aa564027a2c594ca4c31e1c8
0625 13:52:18.206907 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.291857 (+ 84950us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.302394 (+ 10537us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.369093 (+ 66699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141225}
I20250625 13:52:18.369837 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 3249c5c2aa564027a2c594ca4c31e1c8 (table default.Table_248a6183f6d44e43ab1fa4722ec9a52e [id=85870f1c32df485aaef7d845e855b022]) successfully deleted
W20250625 13:52:18.370097 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45d68de75ac84f1a8138c38fd51547cf consensus state has no leader
I20250625 13:52:18.370414 1165 tablet_replica.cc:331] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.370635 1165 raft_consensus.cc:2241] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.370805 1165 raft_consensus.cc:2270] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.371596 1165 ts_tablet_manager.cc:1905] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.372864 1170 tablet_bootstrap.cc:492] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:18.373059 1171 ts_tablet_manager.cc:1428] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20250625 13:52:18.373095 1170 ts_tablet_manager.cc:1397] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.006s
I20250625 13:52:18.373809 1170 raft_consensus.cc:357] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.374051 1170 raft_consensus.cc:383] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.374161 1170 raft_consensus.cc:738] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.374454 1170 consensus_queue.cc:260] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.375123 1170 ts_tablet_manager.cc:1428] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:18.375483 1170 tablet_bootstrap.cc:492] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:18.376196 1171 tablet_bootstrap.cc:492] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:18.382511 2569 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: acquiring CA information for follower catalog manager: success
I20250625 13:52:18.383260 1171 tablet_bootstrap.cc:654] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.391860 1170 tablet_bootstrap.cc:654] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.395207 1166 tablet_bootstrap.cc:492] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:18.396862 2338 raft_consensus.cc:491] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.397024 2338 raft_consensus.cc:513] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.397658 2338 leader_election.cc:290] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.398074 1166 ts_tablet_manager.cc:1397] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20250625 13:52:18.400180 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26820eaba98401da3a8b5348be21fc6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.400319 588 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 38770320fc6541c39e10448978f8a3b0 (table default.Table_6d9870087cdc4dfca8fa438020887d80 [id=43487ee990e640bcbeefc7c142075998]). Expected version 2 got 0
W20250625 13:52:18.400931 851 leader_election.cc:343] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d26820eaba98401da3a8b5348be21fc6
W20250625 13:52:18.401445 849 leader_election.cc:343] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.401561 849 leader_election.cc:304] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.401873 2580 raft_consensus.cc:2747] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.402787 2594 raft_consensus.cc:491] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.402961 2594 raft_consensus.cc:513] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.403915 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eafc402d809245d189f3977162506ef8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.404448 711 leader_election.cc:343] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.405107 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eafc402d809245d189f3977162506ef8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:18.405567 711 leader_election.cc:343] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.405653 711 leader_election.cc:304] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.405848 2548 raft_consensus.cc:2747] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.406045 2594 leader_election.cc:290] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.416075 1166 raft_consensus.cc:357] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.417867 1166 raft_consensus.cc:383] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.417970 1166 raft_consensus.cc:738] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.419093 1165 ts_tablet_manager.cc:1918] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.419211 1165 log.cc:1199] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/4d2d6f05348e43339e7167a9cddca2ba
I20250625 13:52:18.423744 2338 raft_consensus.cc:491] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.423898 2338 raft_consensus.cc:513] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.424574 2338 leader_election.cc:290] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.418447 1166 consensus_queue.cc:260] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.426095 1166 ts_tablet_manager.cc:1428] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
W20250625 13:52:18.426237 851 leader_election.cc:343] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 63d01af6b320489fb99c467c11ed1dc0
I20250625 13:52:18.426834 1171 tablet_bootstrap.cc:492] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.426919 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63d01af6b320489fb99c467c11ed1dc0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.427011 1171 ts_tablet_manager.cc:1397] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.003s
I20250625 13:52:18.427134 1044 raft_consensus.cc:2466] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:18.427569 849 leader_election.cc:304] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.427949 1171 raft_consensus.cc:357] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.428138 1171 raft_consensus.cc:383] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.428218 1171 raft_consensus.cc:738] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.428454 1171 consensus_queue.cc:260] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.428872 2580 raft_consensus.cc:2802] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.428939 2580 raft_consensus.cc:491] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.428997 2580 raft_consensus.cc:3058] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.431249 1166 tablet_bootstrap.cc:492] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:18.434949 2580 raft_consensus.cc:513] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.435560 2580 leader_election.cc:290] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.436182 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 38770320fc6541c39e10448978f8a3b0 (table default.Table_6d9870087cdc4dfca8fa438020887d80 [id=43487ee990e640bcbeefc7c142075998]). Expected version 2 got 0
I20250625 13:52:18.438215 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63d01af6b320489fb99c467c11ed1dc0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:18.438402 1048 raft_consensus.cc:3058] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.438581 1166 tablet_bootstrap.cc:654] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:18.441172 851 leader_election.cc:343] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 63d01af6b320489fb99c467c11ed1dc0
I20250625 13:52:18.441341 2548 raft_consensus.cc:491] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.441453 2548 raft_consensus.cc:513] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.441973 2548 leader_election.cc:290] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.442636 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7efc8df3c8e24020b073a1b61bc9e0a0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:18.443120 711 leader_election.cc:343] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.444654 1171 ts_tablet_manager.cc:1428] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250625 13:52:18.445092 1171 tablet_bootstrap.cc:492] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:18.448251 1171 tablet_bootstrap.cc:654] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.464404 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7efc8df3c8e24020b073a1b61bc9e0a0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.465015 711 leader_election.cc:343] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.465147 711 leader_election.cc:304] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.465510 1165 ts_tablet_manager.cc:1939] T 4d2d6f05348e43339e7167a9cddca2ba P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.467965 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2600471f74cf412b96ad3545d0f94cba" } modify_external_catalogs: true
I20250625 13:52:18.468142 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_17343b483b8e4268a6a3cca32bbcb186" } modify_external_catalogs: true
I20250625 13:52:18.468291 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_17343b483b8e4268a6a3cca32bbcb186" } modify_external_catalogs: true
I20250625 13:52:18.468148 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2600471f74cf412b96ad3545d0f94cba" } modify_external_catalogs: true
I20250625 13:52:18.471728 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c2bc65ae9c22425881fa52f654ed3541"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:18.472344 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2a8a7faddcd5494fa349ad6ecc6aec18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:18.476130 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e6384d02e4dc478baba1218c68d5765b" } new_table_name: "default.Table_233f211583cc4c0b9c9650f430493663" modify_external_catalogs: true
W20250625 13:52:18.485175 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8684eb9104e44227944b21784c54e496 consensus state has no leader
I20250625 13:52:18.485580 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2841cceab45a479bb090cee626e053d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:18.487545 596 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c80b0c592cb94919b710e4e7c96e8e18
I20250625 13:52:18.487882 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9b61484dc29b44ac9d0afd92e1da0ef3" } modify_external_catalogs: true
I20250625 13:52:18.488023 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9b61484dc29b44ac9d0afd92e1da0ef3" } modify_external_catalogs: true
I20250625 13:52:18.491565 1048 raft_consensus.cc:2466] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:18.492172 849 leader_election.cc:304] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.492662 2518 raft_consensus.cc:2802] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:18.492815 2518 raft_consensus.cc:695] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:18.493095 2518 consensus_queue.cc:237] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.511969 2254 ts_tablet_manager.cc:1918] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.512213 2254 log.cc:1199] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/723246e1e6824faf9f9d39c3ea941fe4
I20250625 13:52:18.512583 2254 ts_tablet_manager.cc:1939] T 723246e1e6824faf9f9d39c3ea941fe4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:18.512943 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1285) took 1584 ms. Trace:
I20250625 13:52:18.513134 2254 rpcz_store.cc:276] 0625 13:52:16.928611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.928891 (+ 280us) service_pool.cc:225] Handling call
0625 13:52:18.261714 (+1332823us) ts_tablet_manager.cc:1145] Deleting tablet 723246e1e6824faf9f9d39c3ea941fe4
0625 13:52:18.261725 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.297669 (+ 35944us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.511829 (+214160us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.512899 (+ 1070us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329001}
I20250625 13:52:18.513783 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1287) took 1403 ms. Trace:
I20250625 13:52:18.513888 2254 rpcz_store.cc:276] 0625 13:52:17.110057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.111723 (+ 1666us) service_pool.cc:225] Handling call
0625 13:52:18.513631 (+1401908us) ts_tablet_manager.cc:1145] Deleting tablet 54cc2d0cf95b41d6b36483af5436fcde
0625 13:52:18.513641 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.513742 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401743}
I20250625 13:52:18.514329 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1288) took 1391 ms. Trace:
I20250625 13:52:18.514485 2254 rpcz_store.cc:276] 0625 13:52:17.122541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.122788 (+ 247us) service_pool.cc:225] Handling call
0625 13:52:18.514214 (+1391426us) ts_tablet_manager.cc:1145] Deleting tablet 575043022b1b4ca686775b2aae41a037
0625 13:52:18.514221 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.514289 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391087}
I20250625 13:52:18.514978 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1292) took 1326 ms. Trace:
I20250625 13:52:18.515118 2254 rpcz_store.cc:276] 0625 13:52:17.188843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.191756 (+ 2913us) service_pool.cc:225] Handling call
0625 13:52:18.514864 (+1323108us) ts_tablet_manager.cc:1145] Deleting tablet 6352e917532840ca832e32ddf0747da4
0625 13:52:18.514872 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.514943 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295095}
I20250625 13:52:18.515587 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1293) took 1312 ms. Trace:
I20250625 13:52:18.515681 2254 rpcz_store.cc:276] 0625 13:52:17.203553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.203604 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:18.515480 (+1311876us) ts_tablet_manager.cc:1145] Deleting tablet a6b2f861d63d4299b3e472cf705a1b67
0625 13:52:18.515489 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.515553 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31,"tablet-delete.queue_time_us":1295711}
I20250625 13:52:18.516217 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1297) took 1213 ms. Trace:
I20250625 13:52:18.516480 2254 rpcz_store.cc:276] 0625 13:52:17.302410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.302543 (+ 133us) service_pool.cc:225] Handling call
0625 13:52:18.516090 (+1213547us) ts_tablet_manager.cc:1145] Deleting tablet aef43c4962c747dbb43039d1e706bbd9
0625 13:52:18.516098 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.516175 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210438}
I20250625 13:52:18.516855 2254 tablet_replica.cc:331] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:18.517053 2254 raft_consensus.cc:2241] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.517293 2254 raft_consensus.cc:2270] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.517885 2254 ts_tablet_manager.cc:1905] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.518822 1170 tablet_bootstrap.cc:492] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:18.519030 1170 ts_tablet_manager.cc:1397] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.004s
I20250625 13:52:18.519845 1170 raft_consensus.cc:357] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.520169 1170 raft_consensus.cc:383] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.520295 1170 raft_consensus.cc:738] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.520655 1170 consensus_queue.cc:260] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.523180 1170 ts_tablet_manager.cc:1428] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:18.465534 2594 raft_consensus.cc:2747] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.466377 1166 tablet_bootstrap.cc:492] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:18.523548 1166 ts_tablet_manager.cc:1397] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.000s
I20250625 13:52:18.523571 1170 tablet_bootstrap.cc:492] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:18.524122 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1306) took 1458 ms. Trace:
I20250625 13:52:18.524298 1166 raft_consensus.cc:357] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.524507 1166 raft_consensus.cc:383] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.524577 1166 raft_consensus.cc:738] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.524781 1166 consensus_queue.cc:260] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.525137 1170 tablet_bootstrap.cc:654] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.525233 1166 ts_tablet_manager.cc:1428] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:18.525529 1166 tablet_bootstrap.cc:492] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:18.527148 1166 tablet_bootstrap.cc:654] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.513604 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 723246e1e6824faf9f9d39c3ea941fe4 (table default.Table_49756985b3fc428cbc871659e5aabdc6 [id=1178d7027158455bb95716dd9e102ef9]) successfully deleted
I20250625 13:52:18.524272 1165 rpcz_store.cc:276] 0625 13:52:17.065739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.065778 (+ 39us) service_pool.cc:225] Handling call
0625 13:52:18.370193 (+1304415us) ts_tablet_manager.cc:1145] Deleting tablet 4d2d6f05348e43339e7167a9cddca2ba
0625 13:52:18.370207 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.416029 (+ 45822us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.418999 (+ 2970us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.524098 (+105099us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304258}
I20250625 13:52:18.530252 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 4d2d6f05348e43339e7167a9cddca2ba (table default.Table_6e6214b878ce4ee6b6152eda7b37e76c [id=83f8d808e36c40deb230bff2a04e7c02]) successfully deleted
I20250625 13:52:18.530663 1165 tablet_replica.cc:331] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.530939 1165 raft_consensus.cc:2241] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.531168 1165 raft_consensus.cc:2270] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.532123 1165 ts_tablet_manager.cc:1905] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.536628 1171 tablet_bootstrap.cc:492] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.536821 1171 ts_tablet_manager.cc:1397] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20250625 13:52:18.538038 1171 raft_consensus.cc:357] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.538384 1171 raft_consensus.cc:383] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.538442 1171 raft_consensus.cc:738] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.538609 1171 consensus_queue.cc:260] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.539067 1171 ts_tablet_manager.cc:1428] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:18.539362 1171 tablet_bootstrap.cc:492] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:18.540879 1171 tablet_bootstrap.cc:654] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.550014 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1202) took 3801 ms. Trace:
W20250625 13:52:18.530068 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 54cc2d0cf95b41d6b36483af5436fcde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54cc2d0cf95b41d6b36483af5436fcde already in progress: creating tablet
W20250625 13:52:18.554872 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 575043022b1b4ca686775b2aae41a037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575043022b1b4ca686775b2aae41a037 already in progress: creating tablet
W20250625 13:52:18.555086 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6352e917532840ca832e32ddf0747da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6352e917532840ca832e32ddf0747da4 already in progress: creating tablet
W20250625 13:52:18.555289 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet a6b2f861d63d4299b3e472cf705a1b67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b2f861d63d4299b3e472cf705a1b67 already in progress: creating tablet
W20250625 13:52:18.555514 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet aef43c4962c747dbb43039d1e706bbd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef43c4962c747dbb43039d1e706bbd9 already in progress: creating tablet
W20250625 13:52:18.555968 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2841cceab45a479bb090cee626e053d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:18.557186 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a8a7faddcd5494fa349ad6ecc6aec18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:18.558143 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2bc65ae9c22425881fa52f654ed3541 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.560410 1168 rpcz_store.cc:276] 0625 13:52:14.748228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.748311 (+ 83us) service_pool.cc:225] Handling call
0625 13:52:18.099872 (+3351561us) ts_tablet_manager.cc:1145] Deleting tablet 60ef50a3624b4c79acc803ff0e2ed364
0625 13:52:18.099884 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.198863 (+ 98979us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.261132 (+ 62269us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.549993 (+288861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3351360}
I20250625 13:52:18.561360 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 60ef50a3624b4c79acc803ff0e2ed364 (table default.Table_3e7baa9cc87143b0833a6bc32a193408 [id=d043c74a6cb94efa89ce819d2ef35545]) successfully deleted
I20250625 13:52:18.561825 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1204) took 3803 ms. Trace:
I20250625 13:52:18.566498 1168 rpcz_store.cc:276] 0625 13:52:14.758254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.758306 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:18.561689 (+3803383us) ts_tablet_manager.cc:1145] Deleting tablet 9710609da88d4b42993b3aef02416543
0625 13:52:18.561700 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.561807 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3803212}
I20250625 13:52:18.567525 1166 tablet_bootstrap.cc:492] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:18.567672 1166 ts_tablet_manager.cc:1397] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20250625 13:52:18.568481 1166 raft_consensus.cc:357] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.568662 1166 raft_consensus.cc:383] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.568732 1166 raft_consensus.cc:738] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.569007 1166 consensus_queue.cc:260] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.569365 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_129a669a3cfe4cffbe85f526c5d047d5" } modify_external_catalogs: true
I20250625 13:52:18.569535 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_129a669a3cfe4cffbe85f526c5d047d5" } modify_external_catalogs: true
I20250625 13:52:18.570052 1166 ts_tablet_manager.cc:1428] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:18.570330 1166 tablet_bootstrap.cc:492] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:18.568665 2518 raft_consensus.cc:491] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.571817 2518 raft_consensus.cc:513] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:18.572041 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9710609da88d4b42993b3aef02416543 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9710609da88d4b42993b3aef02416543 already in progress: creating tablet
I20250625 13:52:18.572353 1168 tablet_replica.cc:331] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:18.572438 2518 leader_election.cc:290] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.572813 1168 raft_consensus.cc:2241] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.573031 1168 raft_consensus.cc:2270] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.573046 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:18.573218 778 raft_consensus.cc:2466] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:18.573689 851 leader_election.cc:304] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.573829 1168 ts_tablet_manager.cc:1905] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.574440 2518 raft_consensus.cc:2802] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.574520 2518 raft_consensus.cc:491] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.574573 2518 raft_consensus.cc:3058] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.575059 2338 raft_consensus.cc:491] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.575155 2338 raft_consensus.cc:513] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.575666 2338 leader_election.cc:290] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.576340 1165 ts_tablet_manager.cc:1918] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.576433 1165 log.cc:1199] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/829221b8a79f460d90dd24be6a2d4c43
I20250625 13:52:18.576733 1165 ts_tablet_manager.cc:1939] T 829221b8a79f460d90dd24be6a2d4c43 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.577069 2518 raft_consensus.cc:513] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.577440 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1307) took 1511 ms. Trace:
I20250625 13:52:18.577950 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_08c064ca59a14ff897cf0347dd79584c" } new_table_name: "default.Table_0dfa96866e4941e7a5ed99a74edacbd5" modify_external_catalogs: true
W20250625 13:52:18.578395 851 leader_election.cc:343] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3249c5c2aa564027a2c594ca4c31e1c8
I20250625 13:52:18.578395 2518 leader_election.cc:290] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.578933 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:18.579103 778 raft_consensus.cc:3058] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.581743 778 raft_consensus.cc:2466] T df72a1a461aa44fe80cff6e3870da8e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:18.582115 851 leader_election.cc:304] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.582350 2518 raft_consensus.cc:2802] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:18.582589 2518 raft_consensus.cc:695] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:18.582940 1171 tablet_bootstrap.cc:492] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.577538 1165 rpcz_store.cc:276] 0625 13:52:17.066233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.066274 (+ 41us) service_pool.cc:225] Handling call
0625 13:52:18.530600 (+1464326us) ts_tablet_manager.cc:1145] Deleting tablet 829221b8a79f460d90dd24be6a2d4c43
0625 13:52:18.530610 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.544705 (+ 14095us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.576234 (+ 31529us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.577420 (+ 1186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464159}
I20250625 13:52:18.583202 1171 ts_tablet_manager.cc:1397] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250625 13:52:18.583447 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1309) took 1472 ms. Trace:
W20250625 13:52:18.583941 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05f205733f3d4c81b9e71709c7559660 consensus state has no leader
I20250625 13:52:18.584123 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 829221b8a79f460d90dd24be6a2d4c43 (table default.Table_dbca3ace8835418e8db2cbc1845d9cfb [id=1cd7ba4de230481e8b0c57c1c585fbd2]) successfully deleted
I20250625 13:52:18.584081 1171 raft_consensus.cc:357] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:18.584240 1171 raft_consensus.cc:383] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.584297 1171 raft_consensus.cc:738] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.584254 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e5270cb847db4a57979fd35c13f6fe0b" } new_table_name: "default.Table_9632dc4166a54c6cb2ec923a50271366" modify_external_catalogs: true
I20250625 13:52:18.584476 1171 consensus_queue.cc:260] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:18.585156 1171 ts_tablet_manager.cc:1428] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:18.585047 1165 rpcz_store.cc:276] 0625 13:52:17.110710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.111931 (+ 1221us) service_pool.cc:225] Handling call
0625 13:52:18.583321 (+1471390us) ts_tablet_manager.cc:1145] Deleting tablet 54cc2d0cf95b41d6b36483af5436fcde
0625 13:52:18.583333 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.583429 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471158}
I20250625 13:52:18.585408 1171 tablet_bootstrap.cc:492] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:18.585517 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 54cc2d0cf95b41d6b36483af5436fcde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54cc2d0cf95b41d6b36483af5436fcde already in progress: creating tablet
I20250625 13:52:18.585947 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1310) took 1461 ms. Trace:
I20250625 13:52:18.586185 2338 raft_consensus.cc:491] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.586289 2338 raft_consensus.cc:513] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.586740 1171 tablet_bootstrap.cc:654] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.586863 2338 leader_election.cc:290] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.582880 2518 consensus_queue.cc:237] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.592178 595 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 38770320fc6541c39e10448978f8a3b0 (table default.Table_6d9870087cdc4dfca8fa438020887d80 [id=43487ee990e640bcbeefc7c142075998]). Expected version 2 got 0
I20250625 13:52:18.593397 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_eaa443a9124c4346a760363614d32112"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.593880 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaa443a9124c4346a760363614d32112 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.587324 2594 raft_consensus.cc:491] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.595113 2594 raft_consensus.cc:513] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.595731 2594 leader_election.cc:290] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.596308 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535812d46d6444fda85d9767c6ca7e62" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.596364 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.596479 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35c7bf4272994e9b8a59062b1757f6f5 consensus state has no leader
I20250625 13:52:18.596799 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5e8ec384f9c44f70a2494253304496f4" } new_table_name: "default.Table_752286ff0d3a4092b5dbe3988747505e" modify_external_catalogs: true
W20250625 13:52:18.596846 711 leader_election.cc:343] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.597127 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249c5c2aa564027a2c594ca4c31e1c8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.597445 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.597661 2556 raft_consensus.cc:491] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.597793 2556 raft_consensus.cc:513] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.593808 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 535812d46d6444fda85d9767c6ca7e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.598309 2556 leader_election.cc:290] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.588847 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:18.598546 778 raft_consensus.cc:2466] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:18.598627 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:18.598883 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 3c561fcc70924ea5ab4ab2b569df215a (table default.Table_7ae6d5c541cf45baaeb485b10d6c9f8a [id=13b265cdb0214da085594e59880edfc7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3c561fcc70924ea5ab4ab2b569df215a consensus state has no leader
I20250625 13:52:18.598924 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535812d46d6444fda85d9767c6ca7e62" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.599100 849 leader_election.cc:343] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:18.599423 849 leader_election.cc:343] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.599519 851 leader_election.cc:304] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:18.599817 711 leader_election.cc:343] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.599848 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (table default.Table_e7d5b0a0e4d64a318b47d1476ac90f2b [id=6903813257eb4459a5b89035cae9e85a]). Expected version 1 got 0
I20250625 13:52:18.599920 2338 raft_consensus.cc:2802] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.599912 711 leader_election.cc:304] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.599978 2338 raft_consensus.cc:491] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.600030 2338 raft_consensus.cc:3058] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.600266 2594 raft_consensus.cc:2747] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.600512 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_81625f4255fe4302808c19d2c855b6e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.600993 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81625f4255fe4302808c19d2c855b6e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.602435 2594 raft_consensus.cc:491] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.602552 2594 raft_consensus.cc:513] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:18.602619 2338 raft_consensus.cc:513] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.603070 2594 leader_election.cc:290] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:18.603096 2338 leader_election.cc:290] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.603765 588 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (table default.Table_6d7df5c325b44ea1899f5faa70e7c37d [id=3977ee3968d246b7a8aed1fafd1822b7]). Expected version 2 got 0
I20250625 13:52:18.603783 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a30a35e3639a43c5b078169060824723" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.603873 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3939ef22ea344328721dfba67d933d1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.603946 913 raft_consensus.cc:2466] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:18.604023 915 raft_consensus.cc:2466] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
W20250625 13:52:18.604246 985 leader_election.cc:343] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a30a35e3639a43c5b078169060824723
I20250625 13:52:18.604282 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:18.604413 776 raft_consensus.cc:3058] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.604432 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_eec2951ee3b343b79d196d6f67178517"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:18.604643 711 leader_election.cc:304] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.604740 983 leader_election.cc:304] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.604866 2594 raft_consensus.cc:2802] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250625 13:52:18.604907 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eec2951ee3b343b79d196d6f67178517 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.604933 2594 raft_consensus.cc:491] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.605000 2594 raft_consensus.cc:3058] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.605113 2556 raft_consensus.cc:2802] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.605188 2556 raft_consensus.cc:491] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.605263 2556 raft_consensus.cc:3058] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.607159 776 raft_consensus.cc:2466] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:18.607553 851 leader_election.cc:304] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.599525 849 leader_election.cc:304] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.608060 757 tablet_service.cc:1515] Processing DeleteTablet for tablet e7bd4939714a4359846345ed0521701c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:07 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:18.608091 849 leader_election.cc:343] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:18.608340 849 leader_election.cc:343] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.608742 2580 raft_consensus.cc:2802] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:18.608844 2580 raft_consensus.cc:695] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:18.609061 2580 consensus_queue.cc:237] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.609359 757 tablet_service.cc:1468] Processing CreateTablet for tablet 3f5e13d947d245d7afdd3140614e7e4d (DEFAULT_TABLE table=default.Table_d069adc5ec614e949d7f2286f54e97a5 [id=b54474da2d61474c982983f9bf981b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.609714 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5e13d947d245d7afdd3140614e7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.609709 2204 raft_consensus.cc:2747] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.609840 2204 raft_consensus.cc:491] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.609937 2204 raft_consensus.cc:513] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:18.610407 2204 leader_election.cc:290] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:18.611157 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3f5e13d947d245d7afdd3140614e7e4d (DEFAULT_TABLE table=default.Table_d069adc5ec614e949d7f2286f54e97a5 [id=b54474da2d61474c982983f9bf981b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.610770 2556 raft_consensus.cc:513] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.613222 2254 ts_tablet_manager.cc:1918] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 13:52:18.613189 985 leader_election.cc:343] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a30a35e3639a43c5b078169060824723
I20250625 13:52:18.613330 2254 log.cc:1199] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/68b8c8aa8036454298bf1e81bab91c01
I20250625 13:52:18.613688 2254 ts_tablet_manager.cc:1939] T 68b8c8aa8036454298bf1e81bab91c01 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:18.614050 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1300) took 1251 ms. Trace:
I20250625 13:52:18.614153 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3f5e13d947d245d7afdd3140614e7e4d (DEFAULT_TABLE table=default.Table_d069adc5ec614e949d7f2286f54e97a5 [id=b54474da2d61474c982983f9bf981b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.614152 2254 rpcz_store.cc:276] 0625 13:52:17.362994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.363041 (+ 47us) service_pool.cc:225] Handling call
0625 13:52:18.516796 (+1153755us) ts_tablet_manager.cc:1145] Deleting tablet 68b8c8aa8036454298bf1e81bab91c01
0625 13:52:18.516804 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.569625 (+ 52821us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.613104 (+ 43479us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.613994 (+ 890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153577}
I20250625 13:52:18.614516 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5e13d947d245d7afdd3140614e7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.614348 2556 leader_election.cc:290] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.614820 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:18.614912 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3939ef22ea344328721dfba67d933d1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:18.616497 711 leader_election.cc:343] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.617570 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a30a35e3639a43c5b078169060824723" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:18.617745 915 raft_consensus.cc:3058] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.618290 2594 raft_consensus.cc:513] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:18.619866 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 68b8c8aa8036454298bf1e81bab91c01 (table default.Table_068bedd508da462681c0138ddd8964e3 [id=3a4d573b5eb84c639bf289ec308eb50b]) successfully deleted
I20250625 13:52:18.620924 915 raft_consensus.cc:2466] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
W20250625 13:52:18.621637 851 leader_election.cc:343] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 829221b8a79f460d90dd24be6a2d4c43
I20250625 13:52:18.621666 983 leader_election.cc:304] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.622128 2556 raft_consensus.cc:2802] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
W20250625 13:52:18.622108 849 leader_election.cc:343] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.622155 2594 leader_election.cc:290] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:18.622327 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 9b2a5c8c45554f1cad3e246309f96480 (DEFAULT_TABLE table=default.Table_d87f41700df148f4a9be83b417176332 [id=673eb2e37e63485eabedeb7026ea3b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.622627 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829221b8a79f460d90dd24be6a2d4c43" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.622756 2556 raft_consensus.cc:695] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:18.623135 2556 consensus_queue.cc:237] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.623777 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3939ef22ea344328721dfba67d933d1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:18.624204 711 leader_election.cc:343] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.624275 890 tablet_service.cc:1468] Processing CreateTablet for tablet 9b2a5c8c45554f1cad3e246309f96480 (DEFAULT_TABLE table=default.Table_d87f41700df148f4a9be83b417176332 [id=673eb2e37e63485eabedeb7026ea3b36]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:18.624516 849 leader_election.cc:343] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.624616 849 leader_election.cc:304] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:18.624991 2338 raft_consensus.cc:2747] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.625633 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ae5f1d414794474b2178b431cbc1d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.626051 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2a5c8c45554f1cad3e246309f96480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.626411 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3939ef22ea344328721dfba67d933d1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:18.626984 915 raft_consensus.cc:3058] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.629483 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9b2a5c8c45554f1cad3e246309f96480 (DEFAULT_TABLE table=default.Table_d87f41700df148f4a9be83b417176332 [id=673eb2e37e63485eabedeb7026ea3b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.630128 915 raft_consensus.cc:2466] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:18.630702 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f5e13d947d245d7afdd3140614e7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.631063 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2a5c8c45554f1cad3e246309f96480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.631896 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 2191b836bb3044c89fee13c869d9e07d
I20250625 13:52:18.632311 589 catalog_manager.cc:6652] ReplaceTablet: tablet 2191b836bb3044c89fee13c869d9e07d deleted and replaced by tablet 93d0f3a82b574dd5bb952e9033d15e2e
I20250625 13:52:18.632853 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 391b412dbb9e420c90aa79473bbbe8d7
I20250625 13:52:18.633561 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 45079deff84a41b488ccdde2f5059431
I20250625 13:52:18.634749 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 2191b836bb3044c89fee13c869d9e07d with delete_type TABLET_DATA_DELETED (replaced by tablet 93d0f3a82b574dd5bb952e9033d15e2e) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.634826 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 2191b836bb3044c89fee13c869d9e07d with delete_type TABLET_DATA_DELETED (replaced by tablet 93d0f3a82b574dd5bb952e9033d15e2e) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.634920 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 391b412dbb9e420c90aa79473bbbe8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.635031 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 45079deff84a41b488ccdde2f5059431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.635023 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 391b412dbb9e420c90aa79473bbbe8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.635171 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 45079deff84a41b488ccdde2f5059431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.635771 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3fc03dbe0a19470fbd0fe550159cd232
I20250625 13:52:18.636827 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fc03dbe0a19470fbd0fe550159cd232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.637253 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fc03dbe0a19470fbd0fe550159cd232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.630561 711 leader_election.cc:304] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:18.638468 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 2191b836bb3044c89fee13c869d9e07d with delete_type TABLET_DATA_DELETED (replaced by tablet 93d0f3a82b574dd5bb952e9033d15e2e) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.638631 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 391b412dbb9e420c90aa79473bbbe8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.638767 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fc03dbe0a19470fbd0fe550159cd232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.638767 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 45079deff84a41b488ccdde2f5059431 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.640872 2594 raft_consensus.cc:2802] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:18.641045 2594 raft_consensus.cc:695] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:18.641382 2594 consensus_queue.cc:237] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:18.643402 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b2a5c8c45554f1cad3e246309f96480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.643477 1171 tablet_bootstrap.cc:492] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.643687 1171 ts_tablet_manager.cc:1397] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.001s
I20250625 13:52:18.644475 1171 raft_consensus.cc:357] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.645360 2254 tablet_replica.cc:331] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:18.645849 2254 raft_consensus.cc:2241] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:18.646135 2254 pending_rounds.cc:70] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:18.646251 2254 pending_rounds.cc:77] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520669040332800 op_type: NO_OP noop_request { }
I20250625 13:52:18.646414 2254 raft_consensus.cc:2887] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:18.646553 2254 raft_consensus.cc:2270] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.586072 1165 rpcz_store.cc:276] 0625 13:52:17.124362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.124967 (+ 605us) service_pool.cc:225] Handling call
0625 13:52:18.585827 (+1460860us) ts_tablet_manager.cc:1145] Deleting tablet 575043022b1b4ca686775b2aae41a037
0625 13:52:18.585836 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.585931 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460685}
W20250625 13:52:18.651233 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 575043022b1b4ca686775b2aae41a037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575043022b1b4ca686775b2aae41a037 already in progress: creating tablet
I20250625 13:52:18.652078 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1313) took 1462 ms. Trace:
I20250625 13:52:18.652276 1165 rpcz_store.cc:276] 0625 13:52:17.189851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.191964 (+ 2113us) service_pool.cc:225] Handling call
0625 13:52:18.651930 (+1459966us) ts_tablet_manager.cc:1145] Deleting tablet 6352e917532840ca832e32ddf0747da4
0625 13:52:18.651943 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.652056 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459696}
W20250625 13:52:18.652797 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6352e917532840ca832e32ddf0747da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6352e917532840ca832e32ddf0747da4 already in progress: creating tablet
I20250625 13:52:18.653232 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1314) took 1447 ms. Trace:
I20250625 13:52:18.653342 1165 rpcz_store.cc:276] 0625 13:52:17.205484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.206077 (+ 593us) service_pool.cc:225] Handling call
0625 13:52:18.653128 (+1447051us) ts_tablet_manager.cc:1145] Deleting tablet a6b2f861d63d4299b3e472cf705a1b67
0625 13:52:18.653137 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.653216 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445519}
W20250625 13:52:18.653827 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a6b2f861d63d4299b3e472cf705a1b67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b2f861d63d4299b3e472cf705a1b67 already in progress: creating tablet
I20250625 13:52:18.656822 750 tablet_service.cc:1468] Processing CreateTablet for tablet 250b0ac801e842cbb39912f80f5e9b82 (DEFAULT_TABLE table=default.Table_bd28610323b44b249068082bd830bd60 [id=b82a612b77024af8a6e1387dd34405bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.657269 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250b0ac801e842cbb39912f80f5e9b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.658238 2254 ts_tablet_manager.cc:1905] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.658308 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 585955810cc1437ab5d7b99f6cc66859 (DEFAULT_TABLE table=default.Table_250cc469d51d495287ef1ff5cd5467e4 [id=e781ec55834d48ba8c465a5002ab310a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.658661 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585955810cc1437ab5d7b99f6cc66859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.662886 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1318) took 1361 ms. Trace:
I20250625 13:52:18.665194 1170 tablet_bootstrap.cc:492] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:18.665417 2548 raft_consensus.cc:491] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.665549 2548 raft_consensus.cc:513] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.665643 749 tablet_service.cc:1468] Processing CreateTablet for tablet 585955810cc1437ab5d7b99f6cc66859 (DEFAULT_TABLE table=default.Table_250cc469d51d495287ef1ff5cd5467e4 [id=e781ec55834d48ba8c465a5002ab310a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.666029 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585955810cc1437ab5d7b99f6cc66859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.666121 2548 leader_election.cc:290] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.663043 1165 rpcz_store.cc:276] 0625 13:52:17.301811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.301867 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:18.662739 (+1360872us) ts_tablet_manager.cc:1145] Deleting tablet aef43c4962c747dbb43039d1e706bbd9
0625 13:52:18.662749 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.662866 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360636}
W20250625 13:52:18.667268 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet aef43c4962c747dbb43039d1e706bbd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef43c4962c747dbb43039d1e706bbd9 already in progress: creating tablet
I20250625 13:52:18.667726 1165 tablet_replica.cc:331] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.668124 1165 raft_consensus.cc:2241] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.667027 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 250b0ac801e842cbb39912f80f5e9b82 (DEFAULT_TABLE table=default.Table_bd28610323b44b249068082bd830bd60 [id=b82a612b77024af8a6e1387dd34405bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.667691 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 1b92b973a0624383be9d357b6abb75f4 (DEFAULT_TABLE table=default.Table_11a3ede4b80d4a44bf251533bede8b96 [id=96000ed2c32549939aabbcd415cca764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.668764 1170 ts_tablet_manager.cc:1397] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.145s user 0.001s sys 0.002s
I20250625 13:52:18.668470 745 tablet_service.cc:1468] Processing CreateTablet for tablet 1b92b973a0624383be9d357b6abb75f4 (DEFAULT_TABLE table=default.Table_11a3ede4b80d4a44bf251533bede8b96 [id=96000ed2c32549939aabbcd415cca764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.669502 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 4a6064ecfccd431e863ac701a13dd0a2 (DEFAULT_TABLE table=default.Table_fe109a1bd53e481d9dc7f6626b5c3759 [id=6cff9064057041a199ae4b2fe73801e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.669759 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b92b973a0624383be9d357b6abb75f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.657407 889 tablet_service.cc:1468] Processing CreateTablet for tablet 250b0ac801e842cbb39912f80f5e9b82 (DEFAULT_TABLE table=default.Table_bd28610323b44b249068082bd830bd60 [id=b82a612b77024af8a6e1387dd34405bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.671958 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250b0ac801e842cbb39912f80f5e9b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.672493 891 tablet_service.cc:1468] Processing CreateTablet for tablet 4a6064ecfccd431e863ac701a13dd0a2 (DEFAULT_TABLE table=default.Table_fe109a1bd53e481d9dc7f6626b5c3759 [id=6cff9064057041a199ae4b2fe73801e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.672788 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6064ecfccd431e863ac701a13dd0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.673036 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 5f7a5ad36bf84acaa7e742138c7aaa6f (DEFAULT_TABLE table=default.Table_39f734a42d3845478cc2f4f5fe79181f [id=1d038a83b5764f99bc9dec7a4e444ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.666564 882 tablet_service.cc:1468] Processing CreateTablet for tablet 585955810cc1437ab5d7b99f6cc66859 (DEFAULT_TABLE table=default.Table_250cc469d51d495287ef1ff5cd5467e4 [id=e781ec55834d48ba8c465a5002ab310a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.675217 1024 tablet_service.cc:1468] Processing CreateTablet for tablet b2d835ce0521436d89438933a9964d47 (DEFAULT_TABLE table=default.Table_1ec3d8755b0d4d0f86175e3f35c24662 [id=079aa2bd25cd43f195377e4997fa5bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.675408 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585955810cc1437ab5d7b99f6cc66859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.675547 896 tablet_service.cc:1468] Processing CreateTablet for tablet 5f7a5ad36bf84acaa7e742138c7aaa6f (DEFAULT_TABLE table=default.Table_39f734a42d3845478cc2f4f5fe79181f [id=1d038a83b5764f99bc9dec7a4e444ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.675981 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7a5ad36bf84acaa7e742138c7aaa6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.676462 890 tablet_service.cc:1468] Processing CreateTablet for tablet b2d835ce0521436d89438933a9964d47 (DEFAULT_TABLE table=default.Table_1ec3d8755b0d4d0f86175e3f35c24662 [id=079aa2bd25cd43f195377e4997fa5bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.676620 749 tablet_service.cc:1468] Processing CreateTablet for tablet 5f7a5ad36bf84acaa7e742138c7aaa6f (DEFAULT_TABLE table=default.Table_39f734a42d3845478cc2f4f5fe79181f [id=1d038a83b5764f99bc9dec7a4e444ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.676779 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d835ce0521436d89438933a9964d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.676934 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7a5ad36bf84acaa7e742138c7aaa6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.679620 753 tablet_service.cc:1468] Processing CreateTablet for tablet b2d835ce0521436d89438933a9964d47 (DEFAULT_TABLE table=default.Table_1ec3d8755b0d4d0f86175e3f35c24662 [id=079aa2bd25cd43f195377e4997fa5bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.679952 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d835ce0521436d89438933a9964d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.680302 750 tablet_service.cc:1468] Processing CreateTablet for tablet 39b99313c3164307bedf8bda32df0b14 (DEFAULT_TABLE table=default.Table_6a20710d360b44daa77b9e6fb6063303 [id=ddb00bff0644441e879a1a4ddb259420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.680635 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39b99313c3164307bedf8bda32df0b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.682169 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d3560cefc3164b1cac1a8768e33041c5 (DEFAULT_TABLE table=default.Table_9925fd56bfad4215a806983cb204acd6 [id=ade9ff84de614eb291bef7d7c8a35507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.682612 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 39b99313c3164307bedf8bda32df0b14 (DEFAULT_TABLE table=default.Table_6a20710d360b44daa77b9e6fb6063303 [id=ddb00bff0644441e879a1a4ddb259420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.683082 894 tablet_service.cc:1468] Processing CreateTablet for tablet 39b99313c3164307bedf8bda32df0b14 (DEFAULT_TABLE table=default.Table_6a20710d360b44daa77b9e6fb6063303 [id=ddb00bff0644441e879a1a4ddb259420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.683123 744 tablet_service.cc:1468] Processing CreateTablet for tablet d3560cefc3164b1cac1a8768e33041c5 (DEFAULT_TABLE table=default.Table_9925fd56bfad4215a806983cb204acd6 [id=ade9ff84de614eb291bef7d7c8a35507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.683385 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39b99313c3164307bedf8bda32df0b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.683432 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3560cefc3164b1cac1a8768e33041c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.683554 1020 tablet_service.cc:1468] Processing CreateTablet for tablet c6db4bfd05794f4fa24731f34481ef85 (DEFAULT_TABLE table=default.Table_0e4bb21a95674c7bb2e602b23226a395 [id=e3489bbcef4c4442ae3703bc98d7304e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.684017 758 tablet_service.cc:1468] Processing CreateTablet for tablet c6db4bfd05794f4fa24731f34481ef85 (DEFAULT_TABLE table=default.Table_0e4bb21a95674c7bb2e602b23226a395 [id=e3489bbcef4c4442ae3703bc98d7304e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.684293 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6db4bfd05794f4fa24731f34481ef85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.686496 878 tablet_service.cc:1468] Processing CreateTablet for tablet d3560cefc3164b1cac1a8768e33041c5 (DEFAULT_TABLE table=default.Table_9925fd56bfad4215a806983cb204acd6 [id=ade9ff84de614eb291bef7d7c8a35507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.686815 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3560cefc3164b1cac1a8768e33041c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.669111 886 tablet_service.cc:1468] Processing CreateTablet for tablet 1b92b973a0624383be9d357b6abb75f4 (DEFAULT_TABLE table=default.Table_11a3ede4b80d4a44bf251533bede8b96 [id=96000ed2c32549939aabbcd415cca764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.687347 879 tablet_service.cc:1468] Processing CreateTablet for tablet c6db4bfd05794f4fa24731f34481ef85 (DEFAULT_TABLE table=default.Table_0e4bb21a95674c7bb2e602b23226a395 [id=e3489bbcef4c4442ae3703bc98d7304e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.687539 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b92b973a0624383be9d357b6abb75f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.690503 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6db4bfd05794f4fa24731f34481ef85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.690827 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 9ec74ef612b54dfdaf29836d49005d68 (DEFAULT_TABLE table=default.Table_d3f4f37b0b30453789b651b77c3b3833 [id=19cdf0d752584848af332cfdc0b491dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.691591 742 tablet_service.cc:1468] Processing CreateTablet for tablet 4a6064ecfccd431e863ac701a13dd0a2 (DEFAULT_TABLE table=default.Table_fe109a1bd53e481d9dc7f6626b5c3759 [id=6cff9064057041a199ae4b2fe73801e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.691962 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6064ecfccd431e863ac701a13dd0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.692653 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9ec74ef612b54dfdaf29836d49005d68 (DEFAULT_TABLE table=default.Table_d3f4f37b0b30453789b651b77c3b3833 [id=19cdf0d752584848af332cfdc0b491dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.692970 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec74ef612b54dfdaf29836d49005d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.699476 1016 tablet_service.cc:1468] Processing CreateTablet for tablet ec8d33d9baf84944a13f3e71d0d0a0a1 (DEFAULT_TABLE table=default.Table_19cabef12f194c91ab805ee4d346fbae [id=0b7eab7b5299492fa0eeebec89cf86fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.700139 1021 tablet_service.cc:1468] Processing CreateTablet for tablet a5e8f00256d44a949d33d4a8b3376da3 (DEFAULT_TABLE table=default.Table_0da0d45b112845ca956fc3f260a6901d [id=939bf3ecf2fc47ce92166a51cf950668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.700342 748 tablet_service.cc:1468] Processing CreateTablet for tablet a5e8f00256d44a949d33d4a8b3376da3 (DEFAULT_TABLE table=default.Table_0da0d45b112845ca956fc3f260a6901d [id=939bf3ecf2fc47ce92166a51cf950668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.700743 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5e8f00256d44a949d33d4a8b3376da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.701953 744 tablet_service.cc:1468] Processing CreateTablet for tablet ec8d33d9baf84944a13f3e71d0d0a0a1 (DEFAULT_TABLE table=default.Table_19cabef12f194c91ab805ee4d346fbae [id=0b7eab7b5299492fa0eeebec89cf86fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.702279 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8d33d9baf84944a13f3e71d0d0a0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.706578 746 tablet_service.cc:1468] Processing CreateTablet for tablet e20749fc53834e18a43593a75a24bb80 (DEFAULT_TABLE table=default.Table_6820cb66764a482097c1ce42f9562f6b [id=9f73d5601f0f4e8ca44f34980a8c7d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.706910 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20749fc53834e18a43593a75a24bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.707173 894 tablet_service.cc:1468] Processing CreateTablet for tablet ec8d33d9baf84944a13f3e71d0d0a0a1 (DEFAULT_TABLE table=default.Table_19cabef12f194c91ab805ee4d346fbae [id=0b7eab7b5299492fa0eeebec89cf86fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.707728 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8d33d9baf84944a13f3e71d0d0a0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.709031 1017 tablet_service.cc:1468] Processing CreateTablet for tablet e20749fc53834e18a43593a75a24bb80 (DEFAULT_TABLE table=default.Table_6820cb66764a482097c1ce42f9562f6b [id=9f73d5601f0f4e8ca44f34980a8c7d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.711621 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 4a5181d545ae4a8fb689eebfde6547bd (DEFAULT_TABLE table=default.Table_dc764e43fd094f22868df876bfcb5654 [id=66312fafc0af4b8bb6feeb87117a30e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.712008 886 tablet_service.cc:1468] Processing CreateTablet for tablet e20749fc53834e18a43593a75a24bb80 (DEFAULT_TABLE table=default.Table_6820cb66764a482097c1ce42f9562f6b [id=9f73d5601f0f4e8ca44f34980a8c7d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.712340 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20749fc53834e18a43593a75a24bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.717808 891 tablet_service.cc:1468] Processing CreateTablet for tablet a5e8f00256d44a949d33d4a8b3376da3 (DEFAULT_TABLE table=default.Table_0da0d45b112845ca956fc3f260a6901d [id=939bf3ecf2fc47ce92166a51cf950668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.718138 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5e8f00256d44a949d33d4a8b3376da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.718523 1018 tablet_service.cc:1468] Processing CreateTablet for tablet bcf0246635b145be8556cca3d67c26c2 (DEFAULT_TABLE table=default.Table_b684a87954be44daa261986a72febecd [id=ec9910fa06aa4d57b4de770322b7f7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.718669 896 tablet_service.cc:1468] Processing CreateTablet for tablet 4a5181d545ae4a8fb689eebfde6547bd (DEFAULT_TABLE table=default.Table_dc764e43fd094f22868df876bfcb5654 [id=66312fafc0af4b8bb6feeb87117a30e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.719149 758 tablet_service.cc:1468] Processing CreateTablet for tablet 9ec74ef612b54dfdaf29836d49005d68 (DEFAULT_TABLE table=default.Table_d3f4f37b0b30453789b651b77c3b3833 [id=19cdf0d752584848af332cfdc0b491dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.719564 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4a5181d545ae4a8fb689eebfde6547bd (DEFAULT_TABLE table=default.Table_dc764e43fd094f22868df876bfcb5654 [id=66312fafc0af4b8bb6feeb87117a30e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.719883 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7531d8f57e9e4b4593b566e0362aa281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.720469 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7531d8f57e9e4b4593b566e0362aa281 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.720579 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec74ef612b54dfdaf29836d49005d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.721686 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9c837b1557cb4c96a2016a326656a84f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.722133 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c837b1557cb4c96a2016a326656a84f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.724970 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ae5f1d414794474b2178b431cbc1d44" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.725993 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5181d545ae4a8fb689eebfde6547bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:18.727178 711 leader_election.cc:343] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.727532 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5181d545ae4a8fb689eebfde6547bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.731877 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e99c95246c7443afa66ac138a1bf6f9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.732470 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e99c95246c7443afa66ac138a1bf6f9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.733759 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_13ab9b8203fe4404afbd33bdd16d06e6" } new_table_name: "default.Table_eb2c1659d4674c8c892d2a81049de520" modify_external_catalogs: true
I20250625 13:52:18.737141 1168 ts_tablet_manager.cc:1918] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.737257 1168 log.cc:1199] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/70959ec681524518a7ec2c9ed4016ad1
I20250625 13:52:18.737536 1168 ts_tablet_manager.cc:1939] T 70959ec681524518a7ec2c9ed4016ad1 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:18.737819 2518 raft_consensus.cc:491] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.737922 2518 raft_consensus.cc:513] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.738232 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1206) took 3977 ms. Trace:
I20250625 13:52:18.738302 1168 rpcz_store.cc:276] 0625 13:52:14.761210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.761265 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:18.572300 (+3811035us) ts_tablet_manager.cc:1145] Deleting tablet 70959ec681524518a7ec2c9ed4016ad1
0625 13:52:18.572311 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.581416 (+ 9105us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.737063 (+155647us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.738208 (+ 1145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3810832}
I20250625 13:52:18.738653 2518 leader_election.cc:290] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.738718 1168 tablet_replica.cc:331] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:18.738901 1168 raft_consensus.cc:2241] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.739017 1168 pending_rounds.cc:70] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:18.739050 1168 pending_rounds.cc:77] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520661704175616 op_type: NO_OP noop_request { }
I20250625 13:52:18.739123 1168 raft_consensus.cc:2887] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:18.739085 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 70959ec681524518a7ec2c9ed4016ad1 (table default.Table_3df36037c9584e4c94d313cdf1402fdd [id=d1dc28e680ae47fe8b0bce895330e353]) successfully deleted
I20250625 13:52:18.739195 1168 raft_consensus.cc:2270] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.739346 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "669de0b7c93a4efea8623dab2e462c8d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:18.739539 778 raft_consensus.cc:2466] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:18.740065 851 leader_election.cc:304] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.740629 2518 raft_consensus.cc:2802] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:18.740711 2518 raft_consensus.cc:491] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:18.740787 2518 raft_consensus.cc:3058] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.741629 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3560cefc3164b1cac1a8768e33041c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.741820 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6db4bfd05794f4fa24731f34481ef85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.741990 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d835ce0521436d89438933a9964d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.742151 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5e8f00256d44a949d33d4a8b3376da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.742323 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5181d545ae4a8fb689eebfde6547bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.742622 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20749fc53834e18a43593a75a24bb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.742766 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf0246635b145be8556cca3d67c26c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.742975 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ec74ef612b54dfdaf29836d49005d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.743199 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6064ecfccd431e863ac701a13dd0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.743443 1165 raft_consensus.cc:2270] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.743937 1165 ts_tablet_manager.cc:1905] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.744751 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7a5ad36bf84acaa7e742138c7aaa6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.745182 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39b99313c3164307bedf8bda32df0b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.745174 2518 raft_consensus.cc:513] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.745747 2518 leader_election.cc:290] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.746114 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "669de0b7c93a4efea8623dab2e462c8d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:18.746268 778 raft_consensus.cc:3058] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:18.747503 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8d33d9baf84944a13f3e71d0d0a0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.747931 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b92b973a0624383be9d357b6abb75f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.749024 778 raft_consensus.cc:2466] T 669de0b7c93a4efea8623dab2e462c8d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:18.750253 851 leader_election.cc:304] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:18.750728 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 69672579a83c41d09ee7c78a6fa7d746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:02 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.750799 2518 raft_consensus.cc:2802] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:18.750998 2518 raft_consensus.cc:695] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:18.751230 2518 consensus_queue.cc:237] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.755879 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2659c0ac66aa4defb4834ba47dd2b6c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.756417 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2659c0ac66aa4defb4834ba47dd2b6c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.767190 1170 raft_consensus.cc:357] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:18.768289 1170 raft_consensus.cc:383] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.768352 1170 raft_consensus.cc:738] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.769748 1170 consensus_queue.cc:260] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:18.772222 1170 ts_tablet_manager.cc:1428] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.103s user 0.002s sys 0.000s
I20250625 13:52:18.772511 1170 tablet_bootstrap.cc:492] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:18.776907 1170 tablet_bootstrap.cc:654] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.785122 1165 ts_tablet_manager.cc:1918] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.786928 1165 log.cc:1199] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/eafc402d809245d189f3977162506ef8
I20250625 13:52:18.787595 1165 ts_tablet_manager.cc:1939] T eafc402d809245d189f3977162506ef8 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.788533 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1332) took 1121 ms. Trace:
I20250625 13:52:18.790175 1165 rpcz_store.cc:276] 0625 13:52:17.667364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.667960 (+ 596us) service_pool.cc:225] Handling call
0625 13:52:18.667676 (+999716us) ts_tablet_manager.cc:1145] Deleting tablet eafc402d809245d189f3977162506ef8
0625 13:52:18.667683 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.777866 (+110183us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.785013 (+ 7147us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.788512 (+ 3499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999523}
I20250625 13:52:18.790711 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet eafc402d809245d189f3977162506ef8 (table default.Table_fac2050c0ed04e3da584967a45644198 [id=e77b05be11de44639e5a6ae4a6367880]) successfully deleted
I20250625 13:52:18.790716 1170 tablet_bootstrap.cc:492] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:18.791083 1170 ts_tablet_manager.cc:1397] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
W20250625 13:52:18.791736 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 14087755cd5744958499f32fd77b655b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14087755cd5744958499f32fd77b655b already in progress: creating tablet
W20250625 13:52:18.792997 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7d7b671a935b43a6a80a37d38e57f230 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7b671a935b43a6a80a37d38e57f230 already in progress: creating tablet
I20250625 13:52:18.793279 1165 tablet_replica.cc:331] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.793495 1165 raft_consensus.cc:2241] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:18.793715 1165 pending_rounds.cc:70] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Trying to abort 1 pending ops.
I20250625 13:52:18.793768 1165 pending_rounds.cc:77] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520670279815168 op_type: NO_OP noop_request { }
I20250625 13:52:18.794595 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b75460e88d6f4692a0992d1601f9783d" } new_table_name: "default.Table_5ca5269279c34d90aaf691d570606afb" modify_external_catalogs: true
I20250625 13:52:18.793859 1165 raft_consensus.cc:2887] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:18.795490 1165 raft_consensus.cc:2270] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.796243 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_25b2b6109cdc4e76b9ef84637b24d955" } modify_external_catalogs: true
I20250625 13:52:18.796274 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2f5270fd3d3e49eeab18022f7aa3f7a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:18.796387 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_25b2b6109cdc4e76b9ef84637b24d955" } modify_external_catalogs: true
W20250625 13:52:18.796784 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f5270fd3d3e49eeab18022f7aa3f7a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:18.797578 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b42693af8f214e34b0326e73b4401f1a consensus state has no leader
I20250625 13:52:18.797883 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_77257139fcd64d738a0fe21defde633a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.798075 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49b81b7f127d4f2c89ecbecb8a419e5b consensus state has no leader
I20250625 13:52:18.798887 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet aafb7eef813f48fc8295fbb30a29cd0d
I20250625 13:52:18.799641 758 tablet_service.cc:1515] Processing DeleteTablet for tablet aafb7eef813f48fc8295fbb30a29cd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.800235 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bc5ae60b26494a4aa4ca38cfdb824db3" } new_table_name: "default.Table_60d2656d772e4e48bc1e3559bf960ab1" modify_external_catalogs: true
I20250625 13:52:18.801093 1165 ts_tablet_manager.cc:1905] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:18.801267 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77257139fcd64d738a0fe21defde633a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.802397 896 tablet_service.cc:1515] Processing DeleteTablet for tablet aafb7eef813f48fc8295fbb30a29cd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.802452 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_b684a87954be44daa261986a72febecd" } new_table_name: "default.Table_738fbb083a6f4399becbb773d53c7cfd" modify_external_catalogs: true
I20250625 13:52:18.807018 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8e5d2fdb4f954bd58324b61be183214e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.807647 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e5d2fdb4f954bd58324b61be183214e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:18.808676 1165 ts_tablet_manager.cc:1918] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:18.808852 1165 log.cc:1199] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/e3939ef22ea344328721dfba67d933d1
I20250625 13:52:18.809250 1165 ts_tablet_manager.cc:1939] T e3939ef22ea344328721dfba67d933d1 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.809450 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet aafb7eef813f48fc8295fbb30a29cd0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.794160 1170 raft_consensus.cc:357] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.809394 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet b7d888e628a94f939ecf2571e60b1948 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:04 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.809693 1170 raft_consensus.cc:383] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.809875 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250b0ac801e842cbb39912f80f5e9b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.810387 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet e3939ef22ea344328721dfba67d933d1 (table default.Table_630c47cb56984efbac74836e412efaa6 [id=8c054766f5424144a4edb39b192451cb]) successfully deleted
I20250625 13:52:18.811128 2594 raft_consensus.cc:491] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:18.813380 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a6220e6861d4f8099258fbfabaf7da8 consensus state has no leader
I20250625 13:52:18.815294 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7e4fbede3f9040709ae4bdc9bbb792ad
I20250625 13:52:18.815572 1171 raft_consensus.cc:383] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.815654 1171 raft_consensus.cc:738] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.815848 1171 consensus_queue.cc:260] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.809867 1170 raft_consensus.cc:738] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.816428 1171 ts_tablet_manager.cc:1428] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.173s user 0.002s sys 0.000s
I20250625 13:52:18.816354 1170 consensus_queue.cc:260] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.816812 1171 tablet_bootstrap.cc:492] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:18.817198 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6cabec757d6749df9a8aae136342febf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cabec757d6749df9a8aae136342febf already in progress: creating tablet
W20250625 13:52:18.817397 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8e4df287c95946e2beb94524073e5ee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e4df287c95946e2beb94524073e5ee2 already in progress: creating tablet
W20250625 13:52:18.817567 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 663588b7e9ee430faa180fec00897e22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663588b7e9ee430faa180fec00897e22 already in progress: creating tablet
I20250625 13:52:18.818562 1171 tablet_bootstrap.cc:654] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.819417 595 catalog_manager.cc:5582] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:18.819837 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (table default.Table_e7d5b0a0e4d64a318b47d1476ac90f2b [id=6903813257eb4459a5b89035cae9e85a]). Expected version 1 got 0
I20250625 13:52:18.820410 2254 ts_tablet_manager.cc:1918] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:18.820283 595 catalog_manager.cc:5582] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:18.820654 595 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet df72a1a461aa44fe80cff6e3870da8e7 (table default.Table_635e90a4272746cab8e8f9802215e784 [id=1ca6396d145c4b68a68e153cf78f8bcc]). Expected version 1 got 0
I20250625 13:52:18.820713 2254 log.cc:1199] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/70959ec681524518a7ec2c9ed4016ad1
I20250625 13:52:18.821280 2254 ts_tablet_manager.cc:1939] T 70959ec681524518a7ec2c9ed4016ad1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:18.822486 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e4fbede3f9040709ae4bdc9bbb792ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:18.826597 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bcf0246635b145be8556cca3d67c26c2 consensus state has no leader
W20250625 13:52:18.828511 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4091a34a3d61408780c5b9231a68110c consensus state has no leader
I20250625 13:52:18.829267 1171 tablet_bootstrap.cc:492] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.829452 1171 ts_tablet_manager.cc:1397] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.007s
I20250625 13:52:18.830247 1171 raft_consensus.cc:357] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.830476 1171 raft_consensus.cc:383] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.830595 1171 raft_consensus.cc:738] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.830875 1171 consensus_queue.cc:260] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.831494 1171 ts_tablet_manager.cc:1428] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:18.831809 1171 tablet_bootstrap.cc:492] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:18.831858 1170 ts_tablet_manager.cc:1428] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20250625 13:52:18.833288 1171 tablet_bootstrap.cc:654] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.837361 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e4fbede3f9040709ae4bdc9bbb792ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.838943 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0220bc5deb894b909ecd95a1724cf43d" } modify_external_catalogs: true
I20250625 13:52:18.839138 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0220bc5deb894b909ecd95a1724cf43d" } modify_external_catalogs: true
I20250625 13:52:18.841374 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_107a64d05b814c5a9cc7b956f38af315" } modify_external_catalogs: true
I20250625 13:52:18.841532 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_107a64d05b814c5a9cc7b956f38af315" } modify_external_catalogs: true
I20250625 13:52:18.842656 1171 tablet_bootstrap.cc:492] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.842832 1171 ts_tablet_manager.cc:1397] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20250625 13:52:18.843602 1171 raft_consensus.cc:357] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.843827 1171 raft_consensus.cc:383] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.843922 1171 raft_consensus.cc:738] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.843955 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6bb83d62643e419cbe5a1ac4c61fea12" } modify_external_catalogs: true
I20250625 13:52:18.844112 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6bb83d62643e419cbe5a1ac4c61fea12" } modify_external_catalogs: true
I20250625 13:52:18.844285 1171 consensus_queue.cc:260] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.845026 1171 ts_tablet_manager.cc:1428] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:18.845321 1171 tablet_bootstrap.cc:492] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:18.846951 1171 tablet_bootstrap.cc:654] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.847361 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet f0a59da2129141a698692c4b31609d1d
I20250625 13:52:18.848459 896 tablet_service.cc:1515] Processing DeleteTablet for tablet f0a59da2129141a698692c4b31609d1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.851464 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e3939ef22ea344328721dfba67d933d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:10 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.852371 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet f0a59da2129141a698692c4b31609d1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.855650 1171 tablet_bootstrap.cc:492] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:18.855854 1171 ts_tablet_manager.cc:1397] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250625 13:52:18.856709 1171 raft_consensus.cc:357] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.856950 1171 raft_consensus.cc:383] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:18.857048 1171 raft_consensus.cc:738] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:18.857308 1171 consensus_queue.cc:260] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.857683 2556 consensus_queue.cc:1035] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:18.857995 1171 ts_tablet_manager.cc:1428] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:18.858296 1171 tablet_bootstrap.cc:492] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:18.860881 1171 tablet_bootstrap.cc:654] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.865389 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 653267b468ef4958b98af0ed6a7bf4d3
I20250625 13:52:18.866180 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet a89db9678dd444d5a4df85baccd62044
I20250625 13:52:18.867971 896 tablet_service.cc:1515] Processing DeleteTablet for tablet a89db9678dd444d5a4df85baccd62044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.868532 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet a89db9678dd444d5a4df85baccd62044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.868773 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 653267b468ef4958b98af0ed6a7bf4d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:18.868825 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 653267b468ef4958b98af0ed6a7bf4d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:18.870077 2523 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:18.872129 2068 ts_manager.cc:194] Registered new tserver with Master: affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.872917 1168 ts_tablet_manager.cc:1905] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.813148 1165 tablet_replica.cc:331] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.875435 2594 raft_consensus.cc:513] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.875543 1165 raft_consensus.cc:2241] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.875720 1165 raft_consensus.cc:2270] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.876001 2594 leader_election.cc:290] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.876209 1165 ts_tablet_manager.cc:1905] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.876435 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "553af7c7326240838f37e3861d4e3357" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.877341 750 tablet_service.cc:1468] Processing CreateTablet for tablet 27aa202a6ec64a0a8a83ad7b8c0add91 (DEFAULT_TABLE table=default.Table_74bb2a80612b44d5b356dcd8ce9a1fde [id=cdf3485426964889826411a48e4b3b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.877352 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 27aa202a6ec64a0a8a83ad7b8c0add91 (DEFAULT_TABLE table=default.Table_74bb2a80612b44d5b356dcd8ce9a1fde [id=cdf3485426964889826411a48e4b3b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.877715 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27aa202a6ec64a0a8a83ad7b8c0add91. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:18.877993 711 leader_election.cc:343] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.878072 886 tablet_service.cc:1468] Processing CreateTablet for tablet bcf0246635b145be8556cca3d67c26c2 (DEFAULT_TABLE table=default.Table_738fbb083a6f4399becbb773d53c7cfd [id=ec9910fa06aa4d57b4de770322b7f7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.878398 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf0246635b145be8556cca3d67c26c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.813724 758 tablet_service.cc:1468] Processing CreateTablet for tablet bcf0246635b145be8556cca3d67c26c2 (DEFAULT_TABLE table=default.Table_b684a87954be44daa261986a72febecd [id=ec9910fa06aa4d57b4de770322b7f7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.818107 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e4fbede3f9040709ae4bdc9bbb792ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.879626 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf0246635b145be8556cca3d67c26c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.882252 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 6af7517f12d14b4d9b953d44c14cafa5 (DEFAULT_TABLE table=default.Table_c75b74f21f5649bdb0ea378dd949c86d [id=7c81b845cc484debb00a0896eb30b6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.882917 891 tablet_service.cc:1468] Processing CreateTablet for tablet 6af7517f12d14b4d9b953d44c14cafa5 (DEFAULT_TABLE table=default.Table_c75b74f21f5649bdb0ea378dd949c86d [id=7c81b845cc484debb00a0896eb30b6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.848727 744 tablet_service.cc:1515] Processing DeleteTablet for tablet f0a59da2129141a698692c4b31609d1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.883596 896 tablet_service.cc:1468] Processing CreateTablet for tablet 27aa202a6ec64a0a8a83ad7b8c0add91 (DEFAULT_TABLE table=default.Table_74bb2a80612b44d5b356dcd8ce9a1fde [id=cdf3485426964889826411a48e4b3b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.883905 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27aa202a6ec64a0a8a83ad7b8c0add91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.884022 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 4505e1385f574803a91da85fa3a7ddf1 (DEFAULT_TABLE table=default.Table_451b836c46da4b9db775fad108a69547 [id=a2e89e1dfadb4184bd1c3379c88f1a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.884986 894 tablet_service.cc:1468] Processing CreateTablet for tablet 4505e1385f574803a91da85fa3a7ddf1 (DEFAULT_TABLE table=default.Table_451b836c46da4b9db775fad108a69547 [id=a2e89e1dfadb4184bd1c3379c88f1a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.885248 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4505e1385f574803a91da85fa3a7ddf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.886441 890 tablet_service.cc:1468] Processing CreateTablet for tablet 654b1213646e42eeab8b97212aeb737e (DEFAULT_TABLE table=default.Table_7cffeb8b84ab45ae9278474f3ed5107f [id=87b9e4b0bf1341d8ab54ec5a0dbd89df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.886495 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 654b1213646e42eeab8b97212aeb737e (DEFAULT_TABLE table=default.Table_7cffeb8b84ab45ae9278474f3ed5107f [id=87b9e4b0bf1341d8ab54ec5a0dbd89df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.886750 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 654b1213646e42eeab8b97212aeb737e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.887321 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3ea096dafe4e47b157fceb66ec8aab (DEFAULT_TABLE table=default.Table_53c0239319f64f789fd158d2a5fb3f12 [id=9a12be6e0ef24132baa25ae8bfaab36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.863471 2548 raft_consensus.cc:491] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.888204 2548 raft_consensus.cc:513] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.888864 2548 leader_election.cc:290] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:18.889490 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 304171310536435db21a6852b31ff208 (DEFAULT_TABLE table=default.Table_7344157e71594339a43c2dd8d6e1fe58 [id=e6c10c206dbf4100bc8bfa250c64517b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.890082 879 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3ea096dafe4e47b157fceb66ec8aab (DEFAULT_TABLE table=default.Table_53c0239319f64f789fd158d2a5fb3f12 [id=9a12be6e0ef24132baa25ae8bfaab36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.890403 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3ea096dafe4e47b157fceb66ec8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.866582 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 653267b468ef4958b98af0ed6a7bf4d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.867049 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a89db9678dd444d5a4df85baccd62044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:18.893951 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 1958d5bce19046a2b4bc607783a45548 (DEFAULT_TABLE table=default.Table_231ba9d7a8614e20b45715e4a09b9437 [id=48ba1774b6ae43c49493d4547f1d8650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.894127 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7bd4939714a4359846345ed0521701c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:18.894161 893 tablet_service.cc:1468] Processing CreateTablet for tablet b573647836e74964b8ca5bc4deb622d4 (DEFAULT_TABLE table=default.Table_3d5208970197492bbdb0abdb8aefb5b7 [id=91acee6a51564584bb02070f28771536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.894511 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b573647836e74964b8ca5bc4deb622d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.894613 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 189cf4dd67b044af999ded7f562091a1 (DEFAULT_TABLE table=default.Table_7ed44586fa6f4aefaf28dbf32f10ed2a [id=d76916420eb94767ba7a1d5fe9f908d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.894958 1012 tablet_service.cc:1468] Processing CreateTablet for tablet b573647836e74964b8ca5bc4deb622d4 (DEFAULT_TABLE table=default.Table_3d5208970197492bbdb0abdb8aefb5b7 [id=91acee6a51564584bb02070f28771536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.895262 1014 tablet_service.cc:1468] Processing CreateTablet for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 (DEFAULT_TABLE table=default.Table_c4c8c1a0038e4cf6add4073a66492474 [id=06275ac87e2e4feab8764f1d28cdd7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.895262 884 tablet_service.cc:1468] Processing CreateTablet for tablet 189cf4dd67b044af999ded7f562091a1 (DEFAULT_TABLE table=default.Table_7ed44586fa6f4aefaf28dbf32f10ed2a [id=d76916420eb94767ba7a1d5fe9f908d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.895598 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189cf4dd67b044af999ded7f562091a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.895612 1028 tablet_service.cc:1468] Processing CreateTablet for tablet a2ca40cbace54eb79ad5160907b6f492 (DEFAULT_TABLE table=default.Table_d942502e3483477a927a158d0a4a3f4d [id=e64c35df9a9640fd90316ddf385725f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.895934 889 tablet_service.cc:1468] Processing CreateTablet for tablet 304171310536435db21a6852b31ff208 (DEFAULT_TABLE table=default.Table_7344157e71594339a43c2dd8d6e1fe58 [id=e6c10c206dbf4100bc8bfa250c64517b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.896219 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304171310536435db21a6852b31ff208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.894161 878 tablet_service.cc:1468] Processing CreateTablet for tablet 1958d5bce19046a2b4bc607783a45548 (DEFAULT_TABLE table=default.Table_231ba9d7a8614e20b45715e4a09b9437 [id=48ba1774b6ae43c49493d4547f1d8650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.897931 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1958d5bce19046a2b4bc607783a45548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.898355 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d476613e99bb488da1d4d46a7fe3ed33 (DEFAULT_TABLE table=default.Table_bdc29ce3c8744673b2c3c09c5f27acd5 [id=2998facb1b8d419c81fcbaad537c7bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.899221 882 tablet_service.cc:1468] Processing CreateTablet for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 (DEFAULT_TABLE table=default.Table_c4c8c1a0038e4cf6add4073a66492474 [id=06275ac87e2e4feab8764f1d28cdd7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.899533 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:18.899297 711 leader_election.cc:343] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.899718 1030 tablet_service.cc:1468] Processing CreateTablet for tablet f016120ef2ff4183b599c4d9e58d0c48 (DEFAULT_TABLE table=default.Table_69c35651e2b2452894ec73fae8eea1bd [id=3954de3444ba494388130f23dbdad5cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.899789 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6af7517f12d14b4d9b953d44c14cafa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.901064 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 7d9e808df3f44fffb85bf144f9e79557 (DEFAULT_TABLE table=default.Table_29f4b702a71c42d38dc12caed1b9e4b0 [id=3b2db2023698426486defcd332d7b109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.902323 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 74ddc8b8a1e24fe58b6217ddbebd2c7c (DEFAULT_TABLE table=default.Table_21ce01063f674155be1463898a5a2ea5 [id=d8ebd49953004ee4b105cc4f1bb0055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.902966 892 tablet_service.cc:1468] Processing CreateTablet for tablet a2ca40cbace54eb79ad5160907b6f492 (DEFAULT_TABLE table=default.Table_d942502e3483477a927a158d0a4a3f4d [id=e64c35df9a9640fd90316ddf385725f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.903415 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 5c5ee7388f6f4c549360cf1aaf3ca0b4 (DEFAULT_TABLE table=default.Table_522317d064944d7f9ab5d53e41f116f8 [id=7d9ef2389c6448b2b025727075a67052]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.903486 887 tablet_service.cc:1468] Processing CreateTablet for tablet d7df8e6d3db1463da68af4968dbc3da1 (DEFAULT_TABLE table=default.Table_c7173c8df6c647dbaf9eb9e4cc44e29d [id=f3cac169d23149da83465017df35c340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.903798 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7df8e6d3db1463da68af4968dbc3da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.904278 877 tablet_service.cc:1468] Processing CreateTablet for tablet d476613e99bb488da1d4d46a7fe3ed33 (DEFAULT_TABLE table=default.Table_bdc29ce3c8744673b2c3c09c5f27acd5 [id=2998facb1b8d419c81fcbaad537c7bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.904636 888 tablet_service.cc:1468] Processing CreateTablet for tablet f016120ef2ff4183b599c4d9e58d0c48 (DEFAULT_TABLE table=default.Table_69c35651e2b2452894ec73fae8eea1bd [id=3954de3444ba494388130f23dbdad5cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.904987 881 tablet_service.cc:1468] Processing CreateTablet for tablet 7d9e808df3f44fffb85bf144f9e79557 (DEFAULT_TABLE table=default.Table_29f4b702a71c42d38dc12caed1b9e4b0 [id=3b2db2023698426486defcd332d7b109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.905478 895 tablet_service.cc:1468] Processing CreateTablet for tablet 1d61f3f35b8643f29a68693e6b75079e (DEFAULT_TABLE table=default.Table_83b5cb8b7d284680a7e3a68341c7d2dd [id=9c555209912c490a9146e063fd31909e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.903499 883 tablet_service.cc:1468] Processing CreateTablet for tablet 74ddc8b8a1e24fe58b6217ddbebd2c7c (DEFAULT_TABLE table=default.Table_21ce01063f674155be1463898a5a2ea5 [id=d8ebd49953004ee4b105cc4f1bb0055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.906013 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ca40cbace54eb79ad5160907b6f492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.911137 1166 tablet_bootstrap.cc:654] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:18.911439 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f016120ef2ff4183b599c4d9e58d0c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.911872 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9e808df3f44fffb85bf144f9e79557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.912242 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d476613e99bb488da1d4d46a7fe3ed33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.912672 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74ddc8b8a1e24fe58b6217ddbebd2c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.913190 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d61f3f35b8643f29a68693e6b75079e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.914502 1165 ts_tablet_manager.cc:1918] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:18.914618 1165 log.cc:1199] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/7efc8df3c8e24020b073a1b61bc9e0a0
I20250625 13:52:18.914899 1165 ts_tablet_manager.cc:1939] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:18.917475 896 tablet_service.cc:1468] Processing CreateTablet for tablet 5c5ee7388f6f4c549360cf1aaf3ca0b4 (DEFAULT_TABLE table=default.Table_522317d064944d7f9ab5d53e41f116f8 [id=7d9ef2389c6448b2b025727075a67052]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.917938 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c5ee7388f6f4c549360cf1aaf3ca0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.918674 1165 tablet_replica.cc:331] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:18.918865 1165 raft_consensus.cc:2241] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:18.919018 1165 raft_consensus.cc:2270] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:18.919520 1165 ts_tablet_manager.cc:1905] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:18.921484 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 7efc8df3c8e24020b073a1b61bc9e0a0 (table default.Table_c6c55b9a2baa4454b3ea4e9beb218714 [id=fe62358f99bf4b89a66fdfdcd4670c6b]) successfully deleted
W20250625 13:52:18.923663 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 83518.8us
I20250625 13:52:18.923776 749 tablet_service.cc:1468] Processing CreateTablet for tablet 4505e1385f574803a91da85fa3a7ddf1 (DEFAULT_TABLE table=default.Table_451b836c46da4b9db775fad108a69547 [id=a2e89e1dfadb4184bd1c3379c88f1a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.924149 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4505e1385f574803a91da85fa3a7ddf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.924204 757 tablet_service.cc:1468] Processing CreateTablet for tablet 304171310536435db21a6852b31ff208 (DEFAULT_TABLE table=default.Table_7344157e71594339a43c2dd8d6e1fe58 [id=e6c10c206dbf4100bc8bfa250c64517b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.924580 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304171310536435db21a6852b31ff208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.924856 746 tablet_service.cc:1468] Processing CreateTablet for tablet 1958d5bce19046a2b4bc607783a45548 (DEFAULT_TABLE table=default.Table_231ba9d7a8614e20b45715e4a09b9437 [id=48ba1774b6ae43c49493d4547f1d8650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.925169 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1958d5bce19046a2b4bc607783a45548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.927570 753 tablet_service.cc:1468] Processing CreateTablet for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 (DEFAULT_TABLE table=default.Table_c4c8c1a0038e4cf6add4073a66492474 [id=06275ac87e2e4feab8764f1d28cdd7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.927707 747 tablet_service.cc:1468] Processing CreateTablet for tablet a2ca40cbace54eb79ad5160907b6f492 (DEFAULT_TABLE table=default.Table_d942502e3483477a927a158d0a4a3f4d [id=e64c35df9a9640fd90316ddf385725f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.927866 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.928191 745 tablet_service.cc:1468] Processing CreateTablet for tablet b573647836e74964b8ca5bc4deb622d4 (DEFAULT_TABLE table=default.Table_3d5208970197492bbdb0abdb8aefb5b7 [id=91acee6a51564584bb02070f28771536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.928292 748 tablet_service.cc:1468] Processing CreateTablet for tablet 189cf4dd67b044af999ded7f562091a1 (DEFAULT_TABLE table=default.Table_7ed44586fa6f4aefaf28dbf32f10ed2a [id=d76916420eb94767ba7a1d5fe9f908d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.927536 740 tablet_service.cc:1468] Processing CreateTablet for tablet 1d61f3f35b8643f29a68693e6b75079e (DEFAULT_TABLE table=default.Table_83b5cb8b7d284680a7e3a68341c7d2dd [id=9c555209912c490a9146e063fd31909e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.942055 742 tablet_service.cc:1468] Processing CreateTablet for tablet 654b1213646e42eeab8b97212aeb737e (DEFAULT_TABLE table=default.Table_7cffeb8b84ab45ae9278474f3ed5107f [id=87b9e4b0bf1341d8ab54ec5a0dbd89df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.942492 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3ea096dafe4e47b157fceb66ec8aab (DEFAULT_TABLE table=default.Table_53c0239319f64f789fd158d2a5fb3f12 [id=9a12be6e0ef24132baa25ae8bfaab36b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.942066 756 tablet_service.cc:1468] Processing CreateTablet for tablet d7df8e6d3db1463da68af4968dbc3da1 (DEFAULT_TABLE table=default.Table_c7173c8df6c647dbaf9eb9e4cc44e29d [id=f3cac169d23149da83465017df35c340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.942093 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ca40cbace54eb79ad5160907b6f492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.946089 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b573647836e74964b8ca5bc4deb622d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.949064 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 654b1213646e42eeab8b97212aeb737e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.949095 739 tablet_service.cc:1468] Processing CreateTablet for tablet d476613e99bb488da1d4d46a7fe3ed33 (DEFAULT_TABLE table=default.Table_bdc29ce3c8744673b2c3c09c5f27acd5 [id=2998facb1b8d419c81fcbaad537c7bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.949450 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d476613e99bb488da1d4d46a7fe3ed33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.951628 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189cf4dd67b044af999ded7f562091a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.953429 758 tablet_service.cc:1468] Processing CreateTablet for tablet 6af7517f12d14b4d9b953d44c14cafa5 (DEFAULT_TABLE table=default.Table_c75b74f21f5649bdb0ea378dd949c86d [id=7c81b845cc484debb00a0896eb30b6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:18.953747 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6af7517f12d14b4d9b953d44c14cafa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.955449 2518 raft_consensus.cc:491] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.955592 2518 raft_consensus.cc:513] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.956197 2518 leader_election.cc:290] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:18.960777 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d61f3f35b8643f29a68693e6b75079e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.963608 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7df8e6d3db1463da68af4968dbc3da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.968884 2471 raft_consensus.cc:491] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:18.969033 2471 raft_consensus.cc:513] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:18.969627 2471 leader_election.cc:290] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:18.970604 985 leader_election.cc:343] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: a29708fb688a4a65b84ae2564a040b4c
W20250625 13:52:18.971624 983 leader_election.cc:343] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: a29708fb688a4a65b84ae2564a040b4c
I20250625 13:52:18.971725 983 leader_election.cc:304] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:18.971673 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 50187.7us
I20250625 13:52:18.972128 2471 raft_consensus.cc:2747] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:18.978593 1168 ts_tablet_manager.cc:1918] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:18.980918 2569 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 13:52:18.989723 595 master_service.cc:948] ReplaceTablet: received request to replace tablet d8257f79a42247b48f2736a545362f44 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:18.991227 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "669de0b7c93a4efea8623dab2e462c8d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.991755 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "669de0b7c93a4efea8623dab2e462c8d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:18.993170 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27aa202a6ec64a0a8a83ad7b8c0add91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.993242 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa1c198dd6541809843244c7cc18194" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.993995 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ae5f1d414794474b2178b431cbc1d44" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.994388 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "553af7c7326240838f37e3861d4e3357" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.994742 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7bd4939714a4359846345ed0521701c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:18.995553 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1958d5bce19046a2b4bc607783a45548. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:18.996491 711 leader_election.cc:343] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.996634 711 leader_election.cc:304] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:18.999297 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189cf4dd67b044af999ded7f562091a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:18.999235 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_34cd80c2997d41c281b3eb2b2c5420c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:18.999425 711 leader_election.cc:343] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:18.999555 711 leader_election.cc:304] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:18.999750 711 raft_consensus.cc:2684] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 13:52:18.999796 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34cd80c2997d41c281b3eb2b2c5420c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.000864 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c4c8c1a0038e4cf6add4073a66492474" } modify_external_catalogs: true
I20250625 13:52:19.001056 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c4c8c1a0038e4cf6add4073a66492474" } modify_external_catalogs: true
I20250625 13:52:19.001019 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_13ac69935792464f8d971aa8bcb9533d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.001471 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13ac69935792464f8d971aa8bcb9533d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:19.001708 711 leader_election.cc:343] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.001842 711 leader_election.cc:304] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:19.002686 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304171310536435db21a6852b31ff208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.004885 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ecdaa626099d4adaa1f78a0ea582bfa8" } new_table_name: "default.Table_3fd8fa584be64d3cbd13d977cc8f9719" modify_external_catalogs: true
I20250625 13:52:19.005904 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b573647836e74964b8ca5bc4deb622d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.006405 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_40bba3ccaa054c1cabdb0e640f19575f" } new_table_name: "default.Table_3038b7ccb4f54d2caecfb61667127b56" modify_external_catalogs: true
I20250625 13:52:19.008119 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_87bc90c36619493d9b01ef2e7bad3069"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.008618 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87bc90c36619493d9b01ef2e7bad3069 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.009123 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.009349 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ca40cbace54eb79ad5160907b6f492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.011606 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d476613e99bb488da1d4d46a7fe3ed33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.015100 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_a04efddf105f4643ab4db1cd333eada5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:19.015647 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6af7517f12d14b4d9b953d44c14cafa5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:19.015720 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a04efddf105f4643ab4db1cd333eada5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.015749 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_b4aa069ccd71406ca696a1efc43fab1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.018599 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4aa069ccd71406ca696a1efc43fab1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.020069 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4505e1385f574803a91da85fa3a7ddf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.024168 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3ea096dafe4e47b157fceb66ec8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.024969 2471 raft_consensus.cc:491] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.025102 2471 raft_consensus.cc:513] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.025895 2471 leader_election.cc:290] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.026331 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:19.026547 776 raft_consensus.cc:2466] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:19.026994 985 leader_election.cc:304] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:52:19.027245 2471 raft_consensus.cc:2802] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:19.027343 2471 raft_consensus.cc:491] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:19.027412 2471 raft_consensus.cc:3058] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.028208 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 654b1213646e42eeab8b97212aeb737e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.028401 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74ddc8b8a1e24fe58b6217ddbebd2c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.029886 2471 raft_consensus.cc:513] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.032073 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c5ee7388f6f4c549360cf1aaf3ca0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.032312 1170 tablet_bootstrap.cc:492] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:19.032725 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9e808df3f44fffb85bf144f9e79557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.034389 1170 tablet_bootstrap.cc:654] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.035413 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f016120ef2ff4183b599c4d9e58d0c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.032066 2471 leader_election.cc:290] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.061949 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 1d61f3f35b8643f29a68693e6b75079e (DEFAULT_TABLE table=default.Table_83b5cb8b7d284680a7e3a68341c7d2dd [id=9c555209912c490a9146e063fd31909e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.062364 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d61f3f35b8643f29a68693e6b75079e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.066525 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1311) took 1430 ms. Trace:
I20250625 13:52:19.066738 2254 rpcz_store.cc:276] 0625 13:52:17.636507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.683231 (+ 46724us) service_pool.cc:225] Handling call
0625 13:52:18.645300 (+962069us) ts_tablet_manager.cc:1145] Deleting tablet 70959ec681524518a7ec2c9ed4016ad1
0625 13:52:18.645313 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.813962 (+168649us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.820294 (+ 6332us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.066500 (+246206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":961437}
I20250625 13:52:19.067466 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 70959ec681524518a7ec2c9ed4016ad1 (table default.Table_3df36037c9584e4c94d313cdf1402fdd [id=d1dc28e680ae47fe8b0bce895330e353]) successfully deleted
I20250625 13:52:19.068048 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1313) took 1198 ms. Trace:
I20250625 13:52:19.068200 2254 rpcz_store.cc:276] 0625 13:52:17.869189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.871401 (+ 2212us) service_pool.cc:225] Handling call
0625 13:52:19.067892 (+1196491us) ts_tablet_manager.cc:1145] Deleting tablet 7d7b671a935b43a6a80a37d38e57f230
0625 13:52:19.067905 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.068025 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196214}
W20250625 13:52:19.068754 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 7d7b671a935b43a6a80a37d38e57f230 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d7b671a935b43a6a80a37d38e57f230 already in progress: creating tablet
I20250625 13:52:19.069232 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1312) took 1201 ms. Trace:
I20250625 13:52:19.069442 2254 rpcz_store.cc:276] 0625 13:52:17.868000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.872547 (+ 4547us) service_pool.cc:225] Handling call
0625 13:52:19.069109 (+1196562us) ts_tablet_manager.cc:1145] Deleting tablet 14087755cd5744958499f32fd77b655b
0625 13:52:19.069121 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.069213 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1182564}
I20250625 13:52:19.069993 2254 tablet_replica.cc:331] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:19.069954 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 14087755cd5744958499f32fd77b655b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14087755cd5744958499f32fd77b655b already in progress: creating tablet
I20250625 13:52:19.083858 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d545aae2b146473aac4fcc9634d4a56a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.084544 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d545aae2b146473aac4fcc9634d4a56a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.087085 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e2bc4df39674dc3be0cd32c68f1ebca with delete_type TABLET_DATA_DELETED (replaced by tablet 67e05cc542384b4b8117a1d9330db24b) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.087579 1029 tablet_service.cc:1468] Processing CreateTablet for tablet d7df8e6d3db1463da68af4968dbc3da1 (DEFAULT_TABLE table=default.Table_c7173c8df6c647dbaf9eb9e4cc44e29d [id=f3cac169d23149da83465017df35c340]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:19.088128 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5395ac846dd649b6885c713026065eec consensus state has no leader
I20250625 13:52:19.088681 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7df8e6d3db1463da68af4968dbc3da1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:19.090073 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a079e985358040e78455ac8cb6e49f16 consensus state has no leader
I20250625 13:52:19.083107 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_065d55252ae74debb8216a63c480d278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.092432 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_065d55252ae74debb8216a63c480d278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.097826 1170 tablet_bootstrap.cc:492] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:19.098001 1170 ts_tablet_manager.cc:1397] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20250625 13:52:19.098776 1170 raft_consensus.cc:357] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.098924 1170 raft_consensus.cc:383] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.098969 1170 raft_consensus.cc:738] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.098914 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5b7b7a3dd6cf4f0a9cc2be1c0664cc96" } modify_external_catalogs: true
I20250625 13:52:19.099061 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5b7b7a3dd6cf4f0a9cc2be1c0664cc96" } modify_external_catalogs: true
I20250625 13:52:19.099186 1170 consensus_queue.cc:260] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.099579 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ba8009758e5749ca8afc871aa72f9430" } new_table_name: "default.Table_5cd8f555071649039166c2f662671c40" modify_external_catalogs: true
I20250625 13:52:19.099716 1170 ts_tablet_manager.cc:1428] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:19.100363 1170 tablet_bootstrap.cc:492] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:19.102048 1170 tablet_bootstrap.cc:654] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.030923 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:19.103780 776 raft_consensus.cc:3058] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.108541 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3ea096dafe4e47b157fceb66ec8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.122483 740 tablet_service.cc:1468] Processing CreateTablet for tablet 7d9e808df3f44fffb85bf144f9e79557 (DEFAULT_TABLE table=default.Table_29f4b702a71c42d38dc12caed1b9e4b0 [id=3b2db2023698426486defcd332d7b109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.124660 776 raft_consensus.cc:2466] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:19.129879 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7191422e30b643f9a6ed910dc4b614a3" } new_table_name: "default.Table_d6a073fafd8a4d70951c67d16e010f29" modify_external_catalogs: true
I20250625 13:52:19.131099 2548 raft_consensus.cc:2747] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:19.131253 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9e808df3f44fffb85bf144f9e79557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.131372 2548 raft_consensus.cc:2747] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:19.131572 2548 raft_consensus.cc:491] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.131671 2548 raft_consensus.cc:513] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.135129 744 tablet_service.cc:1468] Processing CreateTablet for tablet 74ddc8b8a1e24fe58b6217ddbebd2c7c (DEFAULT_TABLE table=default.Table_21ce01063f674155be1463898a5a2ea5 [id=d8ebd49953004ee4b105cc4f1bb0055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.136456 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74ddc8b8a1e24fe58b6217ddbebd2c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.136521 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_2bfc1cc9693a45bd9287cab41f0106f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:19.136700 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a614c3633f2a46c09254c7c49ffba6e0" } new_table_name: "default.Table_f64e48d894c542a59e4dd43e991c5436" modify_external_catalogs: true
W20250625 13:52:19.137164 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bfc1cc9693a45bd9287cab41f0106f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.139078 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_eb12aa64a1cb41b4a47f82c64e510a70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.139940 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb12aa64a1cb41b4a47f82c64e510a70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.141234 749 tablet_service.cc:1468] Processing CreateTablet for tablet f016120ef2ff4183b599c4d9e58d0c48 (DEFAULT_TABLE table=default.Table_69c35651e2b2452894ec73fae8eea1bd [id=3954de3444ba494388130f23dbdad5cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.141659 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f016120ef2ff4183b599c4d9e58d0c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.142575 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_182984398a6649f6a197cca6535637f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:19.145321 2548 leader_election.cc:290] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:19.146267 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf962c47820f45e79e0e70f9208a17ff consensus state has no leader
I20250625 13:52:19.146282 745 tablet_service.cc:1468] Processing CreateTablet for tablet 5c5ee7388f6f4c549360cf1aaf3ca0b4 (DEFAULT_TABLE table=default.Table_522317d064944d7f9ab5d53e41f116f8 [id=7d9ef2389c6448b2b025727075a67052]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.146663 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c5ee7388f6f4c549360cf1aaf3ca0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.147017 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2f5270fd3d3e49eeab18022f7aa3f7a3" } modify_external_catalogs: true
I20250625 13:52:19.147256 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2f5270fd3d3e49eeab18022f7aa3f7a3" } modify_external_catalogs: true
W20250625 13:52:19.149453 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_182984398a6649f6a197cca6535637f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.152695 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a2007e51d424053bc2836dedf46add2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:15 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:19.162209 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39e7c95bd30e403f9cd391f01ccec124 consensus state has no leader
I20250625 13:52:19.162968 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d070a8c36af544c6b3795695527477aa" } modify_external_catalogs: true
I20250625 13:52:19.163614 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d070a8c36af544c6b3795695527477aa" } modify_external_catalogs: true
W20250625 13:52:19.174793 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9def3e1e149a4045a1bb2da76ced8eaf consensus state has no leader
I20250625 13:52:19.175277 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_64d842d3737b421f9832c07a67be66e7" } modify_external_catalogs: true
I20250625 13:52:19.175472 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_64d842d3737b421f9832c07a67be66e7" } modify_external_catalogs: true
I20250625 13:52:19.175967 1165 ts_tablet_manager.cc:1918] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:19.176107 1165 log.cc:1199] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/553af7c7326240838f37e3861d4e3357
I20250625 13:52:19.176441 1165 ts_tablet_manager.cc:1939] T 553af7c7326240838f37e3861d4e3357 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:19.177443 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1340) took 1147 ms. Trace:
I20250625 13:52:19.177557 1165 rpcz_store.cc:276] 0625 13:52:18.029725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.029777 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:18.918615 (+888838us) ts_tablet_manager.cc:1145] Deleting tablet 553af7c7326240838f37e3861d4e3357
0625 13:52:18.918624 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.934061 (+ 15437us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.133946 (+199885us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.177416 (+ 43470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":888667}
I20250625 13:52:19.178020 1165 tablet_replica.cc:331] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:19.178252 1165 raft_consensus.cc:2241] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.178408 1165 raft_consensus.cc:2270] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.179091 1165 ts_tablet_manager.cc:1905] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.180300 775 raft_consensus.cc:1273] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer affa1c3035b940e88ca143be7d35840d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:19.181777 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 553af7c7326240838f37e3861d4e3357 (table default.Table_1b0f553b6f7d471fab8cf8cc41d204c3 [id=772bd28ed2964d3b9d09189c07cc34f0]) successfully deleted
I20250625 13:52:19.193354 2471 raft_consensus.cc:491] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.193549 2471 raft_consensus.cc:513] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.194362 2471 leader_election.cc:290] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.199738 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9be75d03121f4ea2906236ac3a5b0764" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:19.200388 711 leader_election.cc:343] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.201701 1165 ts_tablet_manager.cc:1918] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:19.202158 985 leader_election.cc:304] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:52:19.202356 1165 log.cc:1199] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/9be75d03121f4ea2906236ac3a5b0764
I20250625 13:52:19.202540 2471 raft_consensus.cc:2802] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:19.202800 1165 ts_tablet_manager.cc:1939] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:19.203375 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1341) took 1056 ms. Trace:
W20250625 13:52:18.995708 849 leader_election.cc:343] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:19.209404 985 consensus_peers.cc:487] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbe4046742b34f1ab0b1610b5d6ee01e. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:19.209393 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 213685us
W20250625 13:52:19.209748 985 leader_election.cc:343] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 69672579a83c41d09ee7c78a6fa7d746
I20250625 13:52:19.038913 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:19.210423 913 raft_consensus.cc:3058] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.214136 913 raft_consensus.cc:2466] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:19.214179 2471 raft_consensus.cc:695] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:19.214473 2471 consensus_queue.cc:237] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.043558 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7c6a325d40c4f7ba2f342e39b9437e3" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.215988 915 raft_consensus.cc:2374] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1. Re-sending same reply.
I20250625 13:52:19.146529 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9be75d03121f4ea2906236ac3a5b0764" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.167721 2580 raft_consensus.cc:491] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.216275 2580 raft_consensus.cc:513] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.216835 2580 leader_election.cc:290] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:19.181545 851 leader_election.cc:343] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: afa1c198dd6541809843244c7cc18194
I20250625 13:52:19.217319 2338 consensus_queue.cc:1035] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:19.218369 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69672579a83c41d09ee7c78a6fa7d746" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.218519 914 raft_consensus.cc:2466] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:19.220283 983 leader_election.cc:304] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:19.220633 851 leader_election.cc:343] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: eafc402d809245d189f3977162506ef8
I20250625 13:52:19.221014 2204 consensus_queue.cc:1035] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:19.208091 2626 raft_consensus.cc:491] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:19.216616 711 leader_election.cc:343] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.226101 711 leader_election.cc:304] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:19.226352 711 raft_consensus.cc:2684] T 9be75d03121f4ea2906236ac3a5b0764 P 5c1aa07ab6bc4b0089e46951240a37c9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 13:52:19.227571 2626 raft_consensus.cc:513] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.228058 1168 log.cc:1199] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/37e4bd4aaab143ffa65a668a952ae345
I20250625 13:52:19.229166 2626 leader_election.cc:290] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.229261 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d51b886302149839efc6f547b944639" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.220748 1166 tablet_bootstrap.cc:492] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
W20250625 13:52:19.229825 711 leader_election.cc:343] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.230070 1166 ts_tablet_manager.cc:1397] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.660s user 0.016s sys 0.029s
I20250625 13:52:19.231029 1166 raft_consensus.cc:357] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.231339 1166 raft_consensus.cc:383] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.231459 1166 raft_consensus.cc:738] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.231843 1166 consensus_queue.cc:260] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.233503 1168 ts_tablet_manager.cc:1939] T 37e4bd4aaab143ffa65a668a952ae345 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:19.235203 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1207) took 4469 ms. Trace:
I20250625 13:52:19.235693 1166 ts_tablet_manager.cc:1428] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 13:52:19.237972 1170 tablet_bootstrap.cc:492] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:19.238189 1170 ts_tablet_manager.cc:1397] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.007s
I20250625 13:52:19.238369 1166 tablet_bootstrap.cc:492] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:19.232118 1165 rpcz_store.cc:276] 0625 13:52:18.146616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.146689 (+ 73us) service_pool.cc:225] Handling call
0625 13:52:19.177958 (+1031269us) ts_tablet_manager.cc:1145] Deleting tablet 9be75d03121f4ea2906236ac3a5b0764
0625 13:52:19.177971 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.190878 (+ 12907us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.201591 (+ 10713us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.203353 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031026}
I20250625 13:52:19.239279 1170 raft_consensus.cc:357] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.239578 1170 raft_consensus.cc:383] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.239684 1170 raft_consensus.cc:738] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.239696 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 9be75d03121f4ea2906236ac3a5b0764 (table default.Table_9b8e7cde4c874918a309446de3d87d66 [id=8884bbb0de4d43bc89a4b9fc3d356c4d]) successfully deleted
I20250625 13:52:19.240021 1170 consensus_queue.cc:260] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.241043 2636 raft_consensus.cc:2802] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:19.241473 2636 raft_consensus.cc:491] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:19.241734 2636 raft_consensus.cc:3058] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.244488 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d51b886302149839efc6f547b944639" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:19.245126 711 leader_election.cc:343] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.245250 711 leader_election.cc:304] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:19.245777 2594 raft_consensus.cc:2747] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:19.246134 1128 log.cc:927] Time spent T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Append to log took a long time: real 0.053s user 0.001s sys 0.001s
I20250625 13:52:19.249749 2636 raft_consensus.cc:513] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.252372 2254 raft_consensus.cc:2241] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:19.253206 2254 raft_consensus.cc:2270] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.254665 2254 ts_tablet_manager.cc:1905] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:19.264122 985 leader_election.cc:343] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 69672579a83c41d09ee7c78a6fa7d746
I20250625 13:52:19.264547 1168 rpcz_store.cc:276] 0625 13:52:14.765964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.766034 (+ 70us) service_pool.cc:225] Handling call
0625 13:52:18.738669 (+3972635us) ts_tablet_manager.cc:1145] Deleting tablet 37e4bd4aaab143ffa65a668a952ae345
0625 13:52:18.738681 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:18.975469 (+236788us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:18.978481 (+ 3012us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.235182 (+256701us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3972461}
I20250625 13:52:19.264910 1168 tablet_replica.cc:331] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:19.265100 1168 raft_consensus.cc:2241] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.265285 1168 raft_consensus.cc:2270] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.265811 1168 ts_tablet_manager.cc:1905] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.265997 2636 leader_election.cc:290] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.266965 1166 tablet_bootstrap.cc:654] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.267217 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 37e4bd4aaab143ffa65a668a952ae345 (table default.Table_3ac6f01366104d75a5bf908bc40c3f4f [id=8a9015750b254a9ea6a48a9510aeddf4]) successfully deleted
I20250625 13:52:19.247903 1171 tablet_bootstrap.cc:492] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:19.264408 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69672579a83c41d09ee7c78a6fa7d746" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:19.267939 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1342) took 1002 ms. Trace:
I20250625 13:52:19.268980 914 raft_consensus.cc:3058] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.269156 1171 ts_tablet_manager.cc:1397] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.411s user 0.050s sys 0.093s
I20250625 13:52:19.269081 1165 rpcz_store.cc:276] 0625 13:52:18.265644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.265706 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:19.240086 (+974380us) ts_tablet_manager.cc:1145] Deleting tablet d92fe17f576e491eb189a10ca72a7e09
0625 13:52:19.240100 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.267919 (+ 27819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":974165}
W20250625 13:52:19.269590 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d92fe17f576e491eb189a10ca72a7e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d92fe17f576e491eb189a10ca72a7e09 already in progress: creating tablet
I20250625 13:52:19.270254 1170 ts_tablet_manager.cc:1428] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.032s user 0.000s sys 0.004s
I20250625 13:52:19.271917 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1343) took 1005 ms. Trace:
I20250625 13:52:19.272048 1165 rpcz_store.cc:276] 0625 13:52:18.266348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.266393 (+ 45us) service_pool.cc:225] Handling call
0625 13:52:19.269907 (+1003514us) ts_tablet_manager.cc:1145] Deleting tablet 6c89a899837f40a8ac9eed9f1d68e420
0625 13:52:19.269917 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.271899 (+ 1982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003363}
W20250625 13:52:19.272456 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 6c89a899837f40a8ac9eed9f1d68e420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c89a899837f40a8ac9eed9f1d68e420 already in progress: creating tablet
I20250625 13:52:19.272984 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1344) took 1002 ms. Trace:
I20250625 13:52:19.273084 1165 rpcz_store.cc:276] 0625 13:52:18.270685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.270744 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:19.272810 (+1002066us) ts_tablet_manager.cc:1145] Deleting tablet 8a9b6e6e1f2d41288a958115dc18d4fb
0625 13:52:19.272819 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.272970 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001914}
W20250625 13:52:19.273506 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8a9b6e6e1f2d41288a958115dc18d4fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a9b6e6e1f2d41288a958115dc18d4fb already in progress: creating tablet
W20250625 13:52:19.274223 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 8dd0d216fb394f31afa4478ad2097fe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dd0d216fb394f31afa4478ad2097fe5 already in progress: creating tablet
I20250625 13:52:19.274590 1165 tablet_replica.cc:331] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:19.274798 1165 raft_consensus.cc:2241] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.274973 1165 raft_consensus.cc:2270] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.275570 1165 ts_tablet_manager.cc:1905] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.276223 1170 tablet_bootstrap.cc:492] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:19.278395 1170 tablet_bootstrap.cc:654] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.279367 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6dab379b48944304ad9688bb94fd5314"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.279896 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dab379b48944304ad9688bb94fd5314 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.282152 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bee90e8e0f4a4c64bb7459e3a603370c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.282642 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bee90e8e0f4a4c64bb7459e3a603370c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.294332 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_30db53c6b4cd4ba785ca5b2bdf0d23c0" } new_table_name: "default.Table_df04995f971a4d368744f01a326ca229" modify_external_catalogs: true
I20250625 13:52:19.296556 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6d7df5c325b44ea1899f5faa70e7c37d" } new_table_name: "default.Table_727a4473adab487fa65f094237ec794b" modify_external_catalogs: true
I20250625 13:52:19.304308 1171 raft_consensus.cc:357] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:19.305759 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e7bf419440645279046b030e7554893 consensus state has no leader
I20250625 13:52:19.304499 1171 raft_consensus.cc:383] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.305948 1171 raft_consensus.cc:738] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.306164 1171 consensus_queue.cc:260] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.306655 1171 ts_tablet_manager.cc:1428] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.037s user 0.000s sys 0.001s
I20250625 13:52:19.306923 1171 tablet_bootstrap.cc:492] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:19.307092 914 raft_consensus.cc:2466] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:19.308279 983 leader_election.cc:304] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:19.312474 597 catalog_manager.cc:5582] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3). New cstate: current_term: 1 leader_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20250625 13:52:19.312831 597 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (table default.Table_6d7df5c325b44ea1899f5faa70e7c37d [id=3977ee3968d246b7a8aed1fafd1822b7]). Expected version 2 got 0
W20250625 13:52:19.313158 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7c6a325d40c4f7ba2f342e39b9437e3 consensus state has no leader
I20250625 13:52:19.315182 1170 tablet_bootstrap.cc:492] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:19.315402 1170 ts_tablet_manager.cc:1397] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20250625 13:52:19.320617 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7e8396ea40d34c66ab740c66647a6afc
I20250625 13:52:19.321558 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e8396ea40d34c66ab740c66647a6afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:19.324533 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e8396ea40d34c66ab740c66647a6afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.328692 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_5310a42f381546ce905e4a99a1d42909"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.329241 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5310a42f381546ce905e4a99a1d42909 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.331841 1166 tablet_bootstrap.cc:492] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:19.331977 1166 ts_tablet_manager.cc:1397] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.000s
I20250625 13:52:19.322329 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e8396ea40d34c66ab740c66647a6afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.325281 2637 raft_consensus.cc:2802] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:19.332713 1166 raft_consensus.cc:357] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.332867 1166 raft_consensus.cc:383] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.332917 1166 raft_consensus.cc:738] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.333021 2637 raft_consensus.cc:695] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:19.333094 1166 consensus_queue.cc:260] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.333364 2637 consensus_queue.cc:237] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.326752 2254 ts_tablet_manager.cc:1918] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:19.335675 1166 ts_tablet_manager.cc:1428] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:19.335716 2254 log.cc:1199] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/cbe4046742b34f1ab0b1610b5d6ee01e
I20250625 13:52:19.336086 1166 tablet_bootstrap.cc:492] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:19.336162 2254 ts_tablet_manager.cc:1939] T cbe4046742b34f1ab0b1610b5d6ee01e P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:19.336516 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1314) took 1456 ms. Trace:
I20250625 13:52:19.336658 2254 rpcz_store.cc:276] 0625 13:52:17.879728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.879780 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:19.069937 (+1190157us) ts_tablet_manager.cc:1145] Deleting tablet cbe4046742b34f1ab0b1610b5d6ee01e
0625 13:52:19.069950 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.252884 (+182934us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x6160002a1654. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf7f000 00007f18fd291093 00007f18ffd6e137 00007f18ffda01c4 00007f18ffd81c20 00007f18ffd819ad 0000000000a7573c 00007f18f05649df 00007f18f0567cab 00007f18f0567b28
0625 13:52:19.258074 (+ 5190us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.325841 (+ 67767us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.336501 (+ 10660us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"spinlock_wait_cycles":418678016,"tablet-delete.queue_time_us":1183379}
I20250625 13:52:19.337317 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1315) took 1453 ms. Trace:
I20250625 13:52:19.337353 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet cbe4046742b34f1ab0b1610b5d6ee01e (table default.Table_906b9e1a3a8147fb935f7a96968ed294 [id=317ce1c5a5774d6ca8d00ed976bf8615]) successfully deleted
I20250625 13:52:19.337666 1166 tablet_bootstrap.cc:654] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.337446 2254 rpcz_store.cc:276] 0625 13:52:17.884178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.884232 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:19.337198 (+1452966us) ts_tablet_manager.cc:1145] Deleting tablet 6cabec757d6749df9a8aae136342febf
0625 13:52:19.337207 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.337302 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450530}
I20250625 13:52:19.337931 2254 tablet_replica.cc:331] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:19.338200 2254 raft_consensus.cc:2241] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.338403 2254 raft_consensus.cc:2270] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:19.338857 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6cabec757d6749df9a8aae136342febf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cabec757d6749df9a8aae136342febf already in progress: creating tablet
I20250625 13:52:19.341239 1171 tablet_bootstrap.cc:654] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.341414 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_103a8b466dec422ab612c3a87ec82c23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.342160 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_103a8b466dec422ab612c3a87ec82c23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.343288 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (table default.Table_e7d5b0a0e4d64a318b47d1476ac90f2b [id=6903813257eb4459a5b89035cae9e85a]). Expected version 1 got 0
I20250625 13:52:19.343636 592 catalog_manager.cc:5582] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d reported cstate change: term changed from 0 to 1, leader changed from <none> to affa1c3035b940e88ca143be7d35840d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "affa1c3035b940e88ca143be7d35840d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } health_report { overall_health: HEALTHY } } }
I20250625 13:52:19.344009 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet 669de0b7c93a4efea8623dab2e462c8d (table default.Table_8315e81e0d0843819735795f79fa91fb [id=d3092ac7d43a4e9891fe289b304e937c]). Expected version 3 got 0
I20250625 13:52:19.344219 592 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ed3c177bb6dd488aa54bee2a447016ff (table default.Table_c4f1a08649504a7db750c88b65ff985c [id=4631cf687a9c478ba42e9ea3375ca639]). Expected version 1 got 0
I20250625 13:52:19.349190 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet cbd82bb415174dbdbcfafa9d6c3f3aa7
I20250625 13:52:19.343633 656 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 7e8396ea40d34c66ab740c66647a6afc
I20250625 13:52:19.351512 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.352135 879 tablet_service.cc:1515] Processing DeleteTablet for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.352478 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e8396ea40d34c66ab740c66647a6afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.353799 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8700871caf544559b569345d27b54935"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.354285 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8700871caf544559b569345d27b54935 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.356277 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d8257f79a42247b48f2736a545362f44
I20250625 13:52:19.356700 595 catalog_manager.cc:6652] ReplaceTablet: tablet d8257f79a42247b48f2736a545362f44 deleted and replaced by tablet 4be9e4546e8846acb27cd0963cd3ea9d
I20250625 13:52:19.357021 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3ba7e48a0b6d4800b85ad1345163f099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.357458 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ba7e48a0b6d4800b85ad1345163f099 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:19.361001 849 leader_election.cc:343] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.363626 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_12751a3c74f64b27b5baf3fe44befca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.364208 849 leader_election.cc:343] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.364369 849 leader_election.cc:304] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:19.364936 2204 raft_consensus.cc:2747] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:19.365810 849 consensus_peers.cc:487] T ef2b08fba37f4a38880c018ec2d869c8 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:19.366595 849 consensus_peers.cc:487] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:19.349922 745 tablet_service.cc:1515] Processing DeleteTablet for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:18 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:19.351658 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e8396ea40d34c66ab740c66647a6afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:19.359087 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d8257f79a42247b48f2736a545362f44 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9e4546e8846acb27cd0963cd3ea9d) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:19.363788 2643 tablet.cc:1722] T 9fa52e79e00c4f9c97a05c353e11ae2c P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250625 13:52:19.373164 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12751a3c74f64b27b5baf3fe44befca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.374017 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eafc402d809245d189f3977162506ef8" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:19.374830 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet d8257f79a42247b48f2736a545362f44 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9e4546e8846acb27cd0963cd3ea9d) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.375464 879 tablet_service.cc:1515] Processing DeleteTablet for tablet d8257f79a42247b48f2736a545362f44 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9e4546e8846acb27cd0963cd3ea9d) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.371680 1165 ts_tablet_manager.cc:1918] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:19.377110 1165 log.cc:1199] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/535812d46d6444fda85d9767c6ca7e62
I20250625 13:52:19.377483 1165 ts_tablet_manager.cc:1939] T 535812d46d6444fda85d9767c6ca7e62 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:19.380280 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 535812d46d6444fda85d9767c6ca7e62 (table default.Table_e6f5787513b7437ab9fa137510881691 [id=26d6478fa3b646b99c7f2ee572a78be6]) successfully deleted
I20250625 13:52:19.381636 2254 ts_tablet_manager.cc:1905] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.382860 744 tablet_service.cc:1468] Processing CreateTablet for tablet e9baa7c98c6649cebcbc9e562c3adf2b (DEFAULT_TABLE table=default.Table_2659c0ac66aa4defb4834ba47dd2b6c0 [id=b59a69e2fab649978be92b59b2a8f93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.383132 1168 ts_tablet_manager.cc:1918] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:19.383229 1168 log.cc:1199] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/3859f399a7b84de99658380bd6c24ff5
I20250625 13:52:19.383236 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9baa7c98c6649cebcbc9e562c3adf2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.384858 1165 tablet_replica.cc:331] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:19.385087 1165 raft_consensus.cc:2241] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.385222 1165 raft_consensus.cc:2270] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.385500 1170 raft_consensus.cc:357] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.385684 1170 raft_consensus.cc:383] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.385686 1165 ts_tablet_manager.cc:1905] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.385741 1170 raft_consensus.cc:738] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.385989 1170 consensus_queue.cc:260] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.386686 1170 ts_tablet_manager.cc:1428] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250625 13:52:19.389592 851 consensus_peers.cc:487] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63d01af6b320489fb99c467c11ed1dc0. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:19.390403 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8111c3a8d1b24ed6b92fc614772a3593 (DEFAULT_TABLE table=default.Table_8e5d2fdb4f954bd58324b61be183214e [id=7e9c95202197443698941a2b95b0ec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.390813 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8111c3a8d1b24ed6b92fc614772a3593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.332041 1007 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 185) took 1026 ms. Trace:
I20250625 13:52:19.391722 1007 rpcz_store.cc:276] 0625 13:52:18.305516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.308142 (+ 2626us) service_pool.cc:225] Handling call
0625 13:52:18.354366 (+ 46224us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x6180000f6c8c. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd49f8be 00007f18fd29b071 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8
0625 13:52:18.410378 (+ 56012us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x616000154c54. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:18.508312 (+ 97934us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x61800012608c. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd49f8be 00007f18fd29b071 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8
0625 13:52:18.596172 (+ 87860us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x6180001d408c. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd49f8be 00007f18fd29b071 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8
0625 13:52:18.667306 (+ 71134us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x6180001d248c. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fd49f8be 00007f18fd29b071 00007f18ffce13b8 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8
0625 13:52:18.827546 (+160240us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x616000261454. stack: 00007f18f051f336 00007f18f051f0f0 00007f18ef5153ad 0000000000971004 0000000000970fcd 000000000097099a 00007f18fbf4e100 00007f18ffce15af 00007f18fa420a36 00007f18fa420765 00007f18f72c2c7e 00007f18f72c1b5e 00007f18f72c7d95 00007f18f72c86bf 00007f18f72c84e8 0000000000a7573c
0625 13:52:19.332011 (+504465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684649216}
W20250625 13:52:19.396421 849 leader_election.cc:343] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.396611 849 leader_election.cc:304] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:19.396858 2631 raft_consensus.cc:2747] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:19.397294 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e8396ea40d34c66ab740c66647a6afc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.397979 879 tablet_service.cc:1468] Processing CreateTablet for tablet e9baa7c98c6649cebcbc9e562c3adf2b (DEFAULT_TABLE table=default.Table_2659c0ac66aa4defb4834ba47dd2b6c0 [id=b59a69e2fab649978be92b59b2a8f93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.397985 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e9baa7c98c6649cebcbc9e562c3adf2b (DEFAULT_TABLE table=default.Table_2659c0ac66aa4defb4834ba47dd2b6c0 [id=b59a69e2fab649978be92b59b2a8f93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.398330 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9baa7c98c6649cebcbc9e562c3adf2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.404125 887 tablet_service.cc:1468] Processing CreateTablet for tablet 8111c3a8d1b24ed6b92fc614772a3593 (DEFAULT_TABLE table=default.Table_8e5d2fdb4f954bd58324b61be183214e [id=7e9c95202197443698941a2b95b0ec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.404129 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 8111c3a8d1b24ed6b92fc614772a3593 (DEFAULT_TABLE table=default.Table_8e5d2fdb4f954bd58324b61be183214e [id=7e9c95202197443698941a2b95b0ec08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.404466 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8111c3a8d1b24ed6b92fc614772a3593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.404687 1028 tablet_service.cc:1468] Processing CreateTablet for tablet dd2d068842aa40ca93ee1571f37c2724 (DEFAULT_TABLE table=default.Table_77257139fcd64d738a0fe21defde633a [id=eeb0f5342de04ff29a52f286f32d42c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.407212 1168 ts_tablet_manager.cc:1939] T 3859f399a7b84de99658380bd6c24ff5 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:19.408838 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1208) took 4641 ms. Trace:
I20250625 13:52:19.412868 1168 rpcz_store.cc:276] 0625 13:52:14.767235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.767276 (+ 41us) service_pool.cc:225] Handling call
0625 13:52:19.264864 (+4497588us) ts_tablet_manager.cc:1145] Deleting tablet 3859f399a7b84de99658380bd6c24ff5
0625 13:52:19.264874 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.361761 (+ 96887us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.383026 (+ 21265us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.408819 (+ 25793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4497432}
I20250625 13:52:19.415604 1165 ts_tablet_manager.cc:1918] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:19.415829 1165 log.cc:1199] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/e7bd4939714a4359846345ed0521701c
I20250625 13:52:19.419842 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1211) took 4641 ms. Trace:
I20250625 13:52:19.415763 2642 tablet.cc:1722] T 9fa52e79e00c4f9c97a05c353e11ae2c P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:19.420714 1171 tablet_bootstrap.cc:492] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:19.420853 1171 ts_tablet_manager.cc:1397] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.010s
I20250625 13:52:19.421718 1171 raft_consensus.cc:357] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.421916 1171 raft_consensus.cc:383] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.417183 749 tablet_service.cc:1468] Processing CreateTablet for tablet dd2d068842aa40ca93ee1571f37c2724 (DEFAULT_TABLE table=default.Table_77257139fcd64d738a0fe21defde633a [id=eeb0f5342de04ff29a52f286f32d42c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.421965 1171 raft_consensus.cc:738] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.418423 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4ad9e94d787040ebba900ac2ff9d5871 (DEFAULT_TABLE table=default.Table_e99c95246c7443afa66ac138a1bf6f9a [id=2c9eaa20a3ed4ec1a63e7d425d1d94e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.422256 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2d068842aa40ca93ee1571f37c2724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.422194 1171 consensus_queue.cc:260] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.422443 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ad9e94d787040ebba900ac2ff9d5871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.425593 887 tablet_service.cc:1468] Processing CreateTablet for tablet dd2d068842aa40ca93ee1571f37c2724 (DEFAULT_TABLE table=default.Table_77257139fcd64d738a0fe21defde633a [id=eeb0f5342de04ff29a52f286f32d42c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.426028 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 4ad9e94d787040ebba900ac2ff9d5871 (DEFAULT_TABLE table=default.Table_e99c95246c7443afa66ac138a1bf6f9a [id=2c9eaa20a3ed4ec1a63e7d425d1d94e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.426450 1171 ts_tablet_manager.cc:1428] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:19.419975 1168 rpcz_store.cc:276] 0625 13:52:14.778562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.778617 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:19.419710 (+4641093us) ts_tablet_manager.cc:1145] Deleting tablet bbd3200bd1384187978aeb7089ec4e03
0625 13:52:19.419720 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.419827 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4640851}
I20250625 13:52:19.428946 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2d068842aa40ca93ee1571f37c2724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.430657 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 6c920a0a8f1d4329af3b9e5f6a45b45d (DEFAULT_TABLE table=default.Table_81625f4255fe4302808c19d2c855b6e3 [id=4971f9f659ab4f9ab9332056e8f4055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.431437 883 tablet_service.cc:1468] Processing CreateTablet for tablet 6c920a0a8f1d4329af3b9e5f6a45b45d (DEFAULT_TABLE table=default.Table_81625f4255fe4302808c19d2c855b6e3 [id=4971f9f659ab4f9ab9332056e8f4055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.431582 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e1549b9e797a44f89b316a44c5e7ee2e
I20250625 13:52:19.431803 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c920a0a8f1d4329af3b9e5f6a45b45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.431769 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 3859f399a7b84de99658380bd6c24ff5 (table default.Table_cf29f4b4bbb94091affd4440a4b8447c [id=27acc786d57a4c7ab531242dd52f17ab]) successfully deleted
W20250625 13:52:19.432016 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bbd3200bd1384187978aeb7089ec4e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbd3200bd1384187978aeb7089ec4e03 already in progress: creating tablet
I20250625 13:52:19.432145 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a51ce4583eaf46df95387d51d249d550 (DEFAULT_TABLE table=default.Table_9c837b1557cb4c96a2016a326656a84f [id=080f781725ed4e45aa512416d769a961]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.432271 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a24126ce121148d29172d84ff249105f" } new_table_name: "default.Table_efddca4f160e4afc8ed425fed7515412" modify_external_catalogs: true
I20250625 13:52:19.432416 1165 ts_tablet_manager.cc:1939] T e7bd4939714a4359846345ed0521701c P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:19.432487 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet e1549b9e797a44f89b316a44c5e7ee2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.432614 1171 tablet_bootstrap.cc:492] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:19.433132 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1221) took 4559 ms. Trace:
I20250625 13:52:19.433202 1168 rpcz_store.cc:276] 0625 13:52:14.874124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.874374 (+ 250us) service_pool.cc:225] Handling call
0625 13:52:19.426918 (+4552544us) ts_tablet_manager.cc:1145] Deleting tablet d2e8f4e3b45a4065b74485157f31247d
0625 13:52:19.432998 (+ 6080us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.433113 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4552329}
I20250625 13:52:19.433522 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e58128254acf44129f7b4e452ac03fe5
I20250625 13:52:19.433557 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1222) took 4559 ms. Trace:
I20250625 13:52:19.433624 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e1549b9e797a44f89b316a44c5e7ee2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.434082 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c521aad516294312a31b2ebc4e2278b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:19.434398 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet e58128254acf44129f7b4e452ac03fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:19.434535 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c521aad516294312a31b2ebc4e2278b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:19.435446 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d2e8f4e3b45a4065b74485157f31247d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2e8f4e3b45a4065b74485157f31247d already in progress: creating tablet
I20250625 13:52:19.436762 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_685641e78e7d43b0a6432cd7f23aa34e" } new_table_name: "default.Table_4a459ffd7337494cb170b187c2d8fc64" modify_external_catalogs: true
I20250625 13:52:19.436869 1165 tablet_replica.cc:331] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:19.437063 1165 raft_consensus.cc:2241] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.437206 1165 raft_consensus.cc:2270] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.437733 1165 ts_tablet_manager.cc:1905] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.437876 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet fc086d416d0f475987bd67f122487353
I20250625 13:52:19.438446 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_91836dc58519406faa07be52d36ed947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.438884 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91836dc58519406faa07be52d36ed947 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.440404 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6c920a0a8f1d4329af3b9e5f6a45b45d (DEFAULT_TABLE table=default.Table_81625f4255fe4302808c19d2c855b6e3 [id=4971f9f659ab4f9ab9332056e8f4055a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.440794 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c920a0a8f1d4329af3b9e5f6a45b45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.441093 740 tablet_service.cc:1515] Processing DeleteTablet for tablet e1549b9e797a44f89b316a44c5e7ee2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:19.441067 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_08d0c030c2cd4f46ad44c12bfee6842d" } new_table_name: "default.Table_aef664c19f8a47569ac63d700d711236" modify_external_catalogs: true
I20250625 13:52:19.441265 739 tablet_service.cc:1515] Processing DeleteTablet for tablet e58128254acf44129f7b4e452ac03fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:19.441888 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet e7bd4939714a4359846345ed0521701c (table default.Table_3c5334740bcc46cdb576090f3911074a [id=26deb489f6bb45f7b6b51d4a7f8e0c38]) successfully deleted
I20250625 13:52:19.442109 739 tablet_service.cc:1515] Processing DeleteTablet for tablet fc086d416d0f475987bd67f122487353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:19.442986 596 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet 9fa52e79e00c4f9c97a05c353e11ae2c (table default.Table_e7d5b0a0e4d64a318b47d1476ac90f2b [id=6903813257eb4459a5b89035cae9e85a]). Expected version 1 got 0
I20250625 13:52:19.443701 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_30c7d06b103143969bc7540930a5b95b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.444201 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30c7d06b103143969bc7540930a5b95b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.444955 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e58128254acf44129f7b4e452ac03fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.447360 745 tablet_service.cc:1468] Processing CreateTablet for tablet a51ce4583eaf46df95387d51d249d550 (DEFAULT_TABLE table=default.Table_9c837b1557cb4c96a2016a326656a84f [id=080f781725ed4e45aa512416d769a961]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.447785 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a51ce4583eaf46df95387d51d249d550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.454280 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9baa7c98c6649cebcbc9e562c3adf2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.456348 778 raft_consensus.cc:1273] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Refusing update from remote peer 1c008de91eb444f48da2c4f0b2d7a1ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:19.457602 890 tablet_service.cc:1515] Processing DeleteTablet for tablet eafc402d809245d189f3977162506ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:03 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.457851 878 tablet_service.cc:1515] Processing DeleteTablet for tablet fc086d416d0f475987bd67f122487353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.458235 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet fc086d416d0f475987bd67f122487353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.459102 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a51ce4583eaf46df95387d51d249d550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.459333 739 tablet_service.cc:1468] Processing CreateTablet for tablet 73f3a9c1627246af8f020cd75480941e (DEFAULT_TABLE table=default.Table_7531d8f57e9e4b4593b566e0362aa281 [id=39a6012eddee493a8ee508f20f320398]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.459695 750 tablet_service.cc:1468] Processing CreateTablet for tablet a59e6239a59d4f188ee768fe1c2b4df2 (DEFAULT_TABLE table=default.Table_25bef492a0bc4aafa4f94e4917f3ab79 [id=21be6f89b15f4cdea678f1e6c331026d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.459872 740 tablet_service.cc:1468] Processing CreateTablet for tablet dc60ebdbced54d67b6cb712fc04ce85b (DEFAULT_TABLE table=default.Table_c2bc65ae9c22425881fa52f654ed3541 [id=119dad3e0b334948b1bce47f1492cf59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.460330 748 tablet_service.cc:1468] Processing CreateTablet for tablet 27bb0aa3571b432b8bf5b8c70ef06f2f (DEFAULT_TABLE table=default.Table_01a81763e3d0469c86e102451a0e9151 [id=8cfd36aa50bb4a439d978629e7d3da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.460827 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5b2f6b50e30b4bfd979399ef4bb8a7d0 (DEFAULT_TABLE table=default.Table_1df4d80b0f28482e902400731cf43a7c [id=8c0407f80984439c9d17323e21d2a537]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.460920 742 tablet_service.cc:1468] Processing CreateTablet for tablet 618413d2f9e549ae83e86c4eab280bcb (DEFAULT_TABLE table=default.Table_051d5935913847be9d19460f80f73e22 [id=9c37fe47fb954107b657e1fd5f4bdd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.461691 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8111c3a8d1b24ed6b92fc614772a3593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.461874 758 tablet_service.cc:1468] Processing CreateTablet for tablet 6ce10088771c49f7844aa074d3d58a6c (DEFAULT_TABLE table=default.Table_9d0e64e523c24f2ca751d55bb49a7213 [id=70f2caf11f7542feaa2b29dc06b573ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.462097 878 tablet_service.cc:1468] Processing CreateTablet for tablet a51ce4583eaf46df95387d51d249d550 (DEFAULT_TABLE table=default.Table_9c837b1557cb4c96a2016a326656a84f [id=080f781725ed4e45aa512416d769a961]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.462461 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a51ce4583eaf46df95387d51d249d550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.463011 877 tablet_service.cc:1468] Processing CreateTablet for tablet dc60ebdbced54d67b6cb712fc04ce85b (DEFAULT_TABLE table=default.Table_c2bc65ae9c22425881fa52f654ed3541 [id=119dad3e0b334948b1bce47f1492cf59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.463097 890 tablet_service.cc:1468] Processing CreateTablet for tablet 73f3a9c1627246af8f020cd75480941e (DEFAULT_TABLE table=default.Table_7531d8f57e9e4b4593b566e0362aa281 [id=39a6012eddee493a8ee508f20f320398]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.463325 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc60ebdbced54d67b6cb712fc04ce85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.463548 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f3a9c1627246af8f020cd75480941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.464061 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 27bb0aa3571b432b8bf5b8c70ef06f2f (DEFAULT_TABLE table=default.Table_01a81763e3d0469c86e102451a0e9151 [id=8cfd36aa50bb4a439d978629e7d3da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.464112 895 tablet_service.cc:1468] Processing CreateTablet for tablet a59e6239a59d4f188ee768fe1c2b4df2 (DEFAULT_TABLE table=default.Table_25bef492a0bc4aafa4f94e4917f3ab79 [id=21be6f89b15f4cdea678f1e6c331026d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.464394 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59e6239a59d4f188ee768fe1c2b4df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.464838 1025 tablet_service.cc:1468] Processing CreateTablet for tablet dc60ebdbced54d67b6cb712fc04ce85b (DEFAULT_TABLE table=default.Table_c2bc65ae9c22425881fa52f654ed3541 [id=119dad3e0b334948b1bce47f1492cf59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.464974 881 tablet_service.cc:1468] Processing CreateTablet for tablet 27bb0aa3571b432b8bf5b8c70ef06f2f (DEFAULT_TABLE table=default.Table_01a81763e3d0469c86e102451a0e9151 [id=8cfd36aa50bb4a439d978629e7d3da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.465188 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a59e6239a59d4f188ee768fe1c2b4df2 (DEFAULT_TABLE table=default.Table_25bef492a0bc4aafa4f94e4917f3ab79 [id=21be6f89b15f4cdea678f1e6c331026d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.465261 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27bb0aa3571b432b8bf5b8c70ef06f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.465543 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 227023f1cd3a49649cf5b3a08f604811 (DEFAULT_TABLE table=default.Table_deb3b74de40448608a132c85f19617d0 [id=56c7b77fb67b4a8e92765c904dfa831b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.465728 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 618413d2f9e549ae83e86c4eab280bcb (DEFAULT_TABLE table=default.Table_051d5935913847be9d19460f80f73e22 [id=9c37fe47fb954107b657e1fd5f4bdd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.465830 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5b2f6b50e30b4bfd979399ef4bb8a7d0 (DEFAULT_TABLE table=default.Table_1df4d80b0f28482e902400731cf43a7c [id=8c0407f80984439c9d17323e21d2a537]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.465907 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 73f3a9c1627246af8f020cd75480941e (DEFAULT_TABLE table=default.Table_7531d8f57e9e4b4593b566e0362aa281 [id=39a6012eddee493a8ee508f20f320398]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.466109 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 35166d655a334fabb5f59fbebb1cb00f (DEFAULT_TABLE table=default.Table_9998378b240c46bcb121279c983960cc [id=48466b87e16942b59e1a05c811ecfd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.464449 893 tablet_service.cc:1468] Processing CreateTablet for tablet 227023f1cd3a49649cf5b3a08f604811 (DEFAULT_TABLE table=default.Table_deb3b74de40448608a132c85f19617d0 [id=56c7b77fb67b4a8e92765c904dfa831b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.466517 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 227023f1cd3a49649cf5b3a08f604811. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.466782 2636 consensus_queue.cc:1035] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:19.467196 2636 raft_consensus.cc:491] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.467327 2636 raft_consensus.cc:513] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.467901 2636 leader_election.cc:290] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.468197 1011 tablet_service.cc:1468] Processing CreateTablet for tablet c0f17bbc71994878a6e594657dc33fa9 (DEFAULT_TABLE table=default.Table_d15a9f7239f84b448320dfae951349cf [id=70ff0f1aa8614f5c894f0adc3f88bb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.469264 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:19.469832 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 6ce10088771c49f7844aa074d3d58a6c (DEFAULT_TABLE table=default.Table_9d0e64e523c24f2ca751d55bb49a7213 [id=70f2caf11f7542feaa2b29dc06b573ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.474548 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2d068842aa40ca93ee1571f37c2724. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.476532 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 4e52053e98544e8c9196ce1f3a0070c6 (DEFAULT_TABLE table=default.Table_32b73fb2c3b84d13b81857f6d4a460e5 [id=26b2b91707a344ec987d4a9f3b0f0790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.480008 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 9d76d69162374150ad261bbe4854b863 (DEFAULT_TABLE table=default.Table_736e211328b54c2faf37ec980eae8d82 [id=0feadde1af1f419ea0e6e5523cb27b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.480336 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ad9e94d787040ebba900ac2ff9d5871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.481812 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9dd5c72f4dd54e018848cac47fb1ec7e (DEFAULT_TABLE table=default.Table_2841cceab45a479bb090cee626e053d0 [id=88eb47e2080e44d7bfd9d1d60f8d1e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.483544 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e58128254acf44129f7b4e452ac03fe5 (DEFAULT_TABLE table=default.Table_5b7b7a3dd6cf4f0a9cc2be1c0664cc96 [id=e3265ec482ab4f5481474ff4020bb350]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.484638 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d76d69162374150ad261bbe4854b863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.484768 894 tablet_service.cc:1468] Processing CreateTablet for tablet 35166d655a334fabb5f59fbebb1cb00f (DEFAULT_TABLE table=default.Table_9998378b240c46bcb121279c983960cc [id=48466b87e16942b59e1a05c811ecfd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.485138 896 tablet_service.cc:1468] Processing CreateTablet for tablet 5b2f6b50e30b4bfd979399ef4bb8a7d0 (DEFAULT_TABLE table=default.Table_1df4d80b0f28482e902400731cf43a7c [id=8c0407f80984439c9d17323e21d2a537]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.485445 889 tablet_service.cc:1468] Processing CreateTablet for tablet c0f17bbc71994878a6e594657dc33fa9 (DEFAULT_TABLE table=default.Table_d15a9f7239f84b448320dfae951349cf [id=70ff0f1aa8614f5c894f0adc3f88bb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.485719 892 tablet_service.cc:1468] Processing CreateTablet for tablet 6ce10088771c49f7844aa074d3d58a6c (DEFAULT_TABLE table=default.Table_9d0e64e523c24f2ca751d55bb49a7213 [id=70f2caf11f7542feaa2b29dc06b573ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.487443 591 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet ef2b08fba37f4a38880c018ec2d869c8 (table default.Table_3a4e5f10c2ec453a82fb97800a0d37f7 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]). Expected version 2 got 0
I20250625 13:52:19.487452 891 tablet_service.cc:1468] Processing CreateTablet for tablet 4e52053e98544e8c9196ce1f3a0070c6 (DEFAULT_TABLE table=default.Table_32b73fb2c3b84d13b81857f6d4a460e5 [id=26b2b91707a344ec987d4a9f3b0f0790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.487629 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9dd5c72f4dd54e018848cac47fb1ec7e (DEFAULT_TABLE table=default.Table_2841cceab45a479bb090cee626e053d0 [id=88eb47e2080e44d7bfd9d1d60f8d1e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.487761 882 tablet_service.cc:1468] Processing CreateTablet for tablet 9d76d69162374150ad261bbe4854b863 (DEFAULT_TABLE table=default.Table_736e211328b54c2faf37ec980eae8d82 [id=0feadde1af1f419ea0e6e5523cb27b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.488168 884 tablet_service.cc:1468] Processing CreateTablet for tablet 618413d2f9e549ae83e86c4eab280bcb (DEFAULT_TABLE table=default.Table_051d5935913847be9d19460f80f73e22 [id=9c37fe47fb954107b657e1fd5f4bdd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.488569 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27bb0aa3571b432b8bf5b8c70ef06f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.488981 888 tablet_service.cc:1468] Processing CreateTablet for tablet e58128254acf44129f7b4e452ac03fe5 (DEFAULT_TABLE table=default.Table_5b7b7a3dd6cf4f0a9cc2be1c0664cc96 [id=e3265ec482ab4f5481474ff4020bb350]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.493301 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc60ebdbced54d67b6cb712fc04ce85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.495537 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59e6239a59d4f188ee768fe1c2b4df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.499496 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 227023f1cd3a49649cf5b3a08f604811. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:19.504429 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c74adb1322fc4074a5b29dc7b44ceea9 consensus state has no leader
W20250625 13:52:19.505043 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86588eb3c34c4536b14f76132bdf3adf consensus state has no leader
W20250625 13:52:19.505378 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d18f80d9a51f4afbadfa239fe28a92ca consensus state has no leader
I20250625 13:52:19.506109 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 618413d2f9e549ae83e86c4eab280bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.506359 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35166d655a334fabb5f59fbebb1cb00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.507546 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f17bbc71994878a6e594657dc33fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.508980 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3a36675ae59b4414bee46d495834a281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.509521 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a36675ae59b4414bee46d495834a281 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.510102 914 raft_consensus.cc:1273] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Refusing update from remote peer 1c008de91eb444f48da2c4f0b2d7a1ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:52:19.510576 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce10088771c49f7844aa074d3d58a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.510818 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a0aeb065271449cb97100a02d36b89db" } new_table_name: "default.Table_a325f5acf9ca4163bb8aeed9bdda5625" modify_external_catalogs: true
I20250625 13:52:19.511041 1016 tablet_service.cc:1468] Processing CreateTablet for tablet e700f0cef0e64bd8833bb9c09bb424f1 (DEFAULT_TABLE table=default.Table_eaa443a9124c4346a760363614d32112 [id=509889c267d148ebb93b1f74c03ce336]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.511126 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce10088771c49f7844aa074d3d58a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.512010 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_17e307b2ad28486882dab136084b7778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.512409 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17e307b2ad28486882dab136084b7778 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.513783 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "647c97a8ae484c4a8694e777bc62d602" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.516791 2636 consensus_queue.cc:1035] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:19.517190 2636 consensus_queue.cc:1035] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:19.517846 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_995a32a383304336af2581d84d9db1e0" } new_table_name: "default.Table_cd6da2c43ed64f698f68319faa6f7b9d" modify_external_catalogs: true
I20250625 13:52:19.524535 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e700f0cef0e64bd8833bb9c09bb424f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.524577 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd5c72f4dd54e018848cac47fb1ec7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.525108 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58128254acf44129f7b4e452ac03fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.526722 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2f6b50e30b4bfd979399ef4bb8a7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.526752 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d76d69162374150ad261bbe4854b863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.527269 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f17bbc71994878a6e594657dc33fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.527405 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f3a9c1627246af8f020cd75480941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.527664 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35166d655a334fabb5f59fbebb1cb00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.528031 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2f6b50e30b4bfd979399ef4bb8a7d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.529634 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5027b25ff4d946e8880a10a456943c12" } modify_external_catalogs: true
I20250625 13:52:19.529811 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_5027b25ff4d946e8880a10a456943c12" } modify_external_catalogs: true
I20250625 13:52:19.538079 744 tablet_service.cc:1468] Processing CreateTablet for tablet 4e52053e98544e8c9196ce1f3a0070c6 (DEFAULT_TABLE table=default.Table_32b73fb2c3b84d13b81857f6d4a460e5 [id=26b2b91707a344ec987d4a9f3b0f0790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.538309 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9d76d69162374150ad261bbe4854b863 (DEFAULT_TABLE table=default.Table_736e211328b54c2faf37ec980eae8d82 [id=0feadde1af1f419ea0e6e5523cb27b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.538456 743 tablet_service.cc:1468] Processing CreateTablet for tablet 9dd5c72f4dd54e018848cac47fb1ec7e (DEFAULT_TABLE table=default.Table_2841cceab45a479bb090cee626e053d0 [id=88eb47e2080e44d7bfd9d1d60f8d1e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.539613 752 tablet_service.cc:1468] Processing CreateTablet for tablet e700f0cef0e64bd8833bb9c09bb424f1 (DEFAULT_TABLE table=default.Table_eaa443a9124c4346a760363614d32112 [id=509889c267d148ebb93b1f74c03ce336]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.540205 741 tablet_service.cc:1468] Processing CreateTablet for tablet e58128254acf44129f7b4e452ac03fe5 (DEFAULT_TABLE table=default.Table_5b7b7a3dd6cf4f0a9cc2be1c0664cc96 [id=e3265ec482ab4f5481474ff4020bb350]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.433634 1168 rpcz_store.cc:276] 0625 13:52:14.874325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.874637 (+ 312us) service_pool.cc:225] Handling call
0625 13:52:19.433454 (+4558817us) ts_tablet_manager.cc:1145] Deleting tablet 29e223675c7842b383865a270b97ce58
0625 13:52:19.433463 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.433535 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4558689}
I20250625 13:52:19.551486 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cf8d3ee817784b2aa014fc65c502b75c" } modify_external_catalogs: true
I20250625 13:52:19.551688 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_cf8d3ee817784b2aa014fc65c502b75c" } modify_external_catalogs: true
I20250625 13:52:19.551789 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4ad9e94d787040ebba900ac2ff9d5871 (DEFAULT_TABLE table=default.Table_e99c95246c7443afa66ac138a1bf6f9a [id=2c9eaa20a3ed4ec1a63e7d425d1d94e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.555697 754 tablet_service.cc:1468] Processing CreateTablet for tablet 05ad205ee6e647fc87434079c444e300 (DEFAULT_TABLE table=default.Table_b29ecaa200d14dc583380e80f8bae166 [id=aeea02d7ea5f44e2ba8cd6becf4dbba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.556078 751 tablet_service.cc:1468] Processing CreateTablet for tablet ae8af48a422c4a76b2e5440f1b75dfb3 (DEFAULT_TABLE table=default.Table_3ddbb86e81474e9ca79571a9c0372b74 [id=449814f10d87435e9e60bf524bc61f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.556586 755 tablet_service.cc:1468] Processing CreateTablet for tablet e6fdb54fda5d4458959fdea9102b8ced (DEFAULT_TABLE table=default.Table_3348879866c946209d1b8e1d87153e2f [id=2ee8b46428b94860b6c81a59aeb00203]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.560650 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 618413d2f9e549ae83e86c4eab280bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.554347 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1223) took 4679 ms. Trace:
I20250625 13:52:19.570047 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e52053e98544e8c9196ce1f3a0070c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.570330 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ad9e94d787040ebba900ac2ff9d5871. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.576423 1170 tablet_bootstrap.cc:492] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:19.579540 1170 tablet_bootstrap.cc:654] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.587353 591 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_e7d5b0a0e4d64a318b47d1476ac90f2b [id=6903813257eb4459a5b89035cae9e85a] alter complete (version 1)
W20250625 13:52:19.589161 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13e4738b2e204bb1bc53908429537d8b consensus state has no leader
W20250625 13:52:19.590701 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c930ac853a304fbab4be6fb37df5f79a consensus state has no leader
I20250625 13:52:19.591425 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 3775c883f8534c6d90687152b9ca22d1
I20250625 13:52:19.595178 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e52053e98544e8c9196ce1f3a0070c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.595407 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c920a0a8f1d4329af3b9e5f6a45b45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.595600 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd5c72f4dd54e018848cac47fb1ec7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.599408 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58128254acf44129f7b4e452ac03fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.605295 1168 rpcz_store.cc:276] 0625 13:52:14.874934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.874987 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:19.545099 (+4670112us) ts_tablet_manager.cc:1145] Deleting tablet 0bc136dc84de44b49abcdf4100a86137
0625 13:52:19.545135 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.554292 (+ 9157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4669930}
W20250625 13:52:19.608681 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 13e4738b2e204bb1bc53908429537d8b (table default.Table_a325f5acf9ca4163bb8aeed9bdda5625 [id=d0b6108df9b747169ededc903b7f1841], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 13e4738b2e204bb1bc53908429537d8b consensus state has no leader
I20250625 13:52:19.613512 2636 raft_consensus.cc:491] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.613674 2636 raft_consensus.cc:513] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.614334 2636 leader_election.cc:290] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:19.615434 985 leader_election.cc:343] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8e2bc4df39674dc3be0cd32c68f1ebca
I20250625 13:52:19.614874 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 077e8c001cc646559b5268d4a48c0ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.619412 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e2bc4df39674dc3be0cd32c68f1ebca" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.620361 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1224) took 4717 ms. Trace:
I20250625 13:52:19.620628 916 raft_consensus.cc:2466] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:19.627645 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_74e5533b20eb4141ac3503011a2746f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.628346 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74e5533b20eb4141ac3503011a2746f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.628458 983 leader_election.cc:304] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:19.624714 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 05ad205ee6e647fc87434079c444e300 (DEFAULT_TABLE table=default.Table_b29ecaa200d14dc583380e80f8bae166 [id=aeea02d7ea5f44e2ba8cd6becf4dbba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.628954 2638 raft_consensus.cc:2802] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:19.629277 2638 raft_consensus.cc:491] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:19.629362 2638 raft_consensus.cc:3058] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.629827 887 tablet_service.cc:1468] Processing CreateTablet for tablet e700f0cef0e64bd8833bb9c09bb424f1 (DEFAULT_TABLE table=default.Table_eaa443a9124c4346a760363614d32112 [id=509889c267d148ebb93b1f74c03ce336]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.632974 2638 raft_consensus.cc:513] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.633790 880 tablet_service.cc:1468] Processing CreateTablet for tablet 05ad205ee6e647fc87434079c444e300 (DEFAULT_TABLE table=default.Table_b29ecaa200d14dc583380e80f8bae166 [id=aeea02d7ea5f44e2ba8cd6becf4dbba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.633939 885 tablet_service.cc:1468] Processing CreateTablet for tablet e6fdb54fda5d4458959fdea9102b8ced (DEFAULT_TABLE table=default.Table_3348879866c946209d1b8e1d87153e2f [id=2ee8b46428b94860b6c81a59aeb00203]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.633896 1168 rpcz_store.cc:276] 0625 13:52:14.888611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:14.888668 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:19.605684 (+4717016us) ts_tablet_manager.cc:1145] Deleting tablet 1b2914ce9b4c40a0bc00d0dd5ac72221
0625 13:52:19.605697 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.605801 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4716795}
I20250625 13:52:19.634511 2638 leader_election.cc:290] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:19.634925 985 leader_election.cc:343] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 8e2bc4df39674dc3be0cd32c68f1ebca
W20250625 13:52:19.636960 849 consensus_peers.cc:487] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:19.638298 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ae8af48a422c4a76b2e5440f1b75dfb3 (DEFAULT_TABLE table=default.Table_3ddbb86e81474e9ca79571a9c0372b74 [id=449814f10d87435e9e60bf524bc61f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.638820 2573 consensus_queue.cc:1035] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:19.635730 1014 tablet_service.cc:1468] Processing CreateTablet for tablet e6fdb54fda5d4458959fdea9102b8ced (DEFAULT_TABLE table=default.Table_3348879866c946209d1b8e1d87153e2f [id=2ee8b46428b94860b6c81a59aeb00203]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.644273 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3f5dc7a3e577405391f6d3ec22c932ff" } modify_external_catalogs: true
I20250625 13:52:19.644616 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c95698028b9241c880fe126c21813367" } modify_external_catalogs: true
I20250625 13:52:19.644870 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ee5d5978aaf540b8bc77e7c4bbe55258" } modify_external_catalogs: true
I20250625 13:52:19.649534 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 7e8396ea40d34c66ab740c66647a6afc (DEFAULT_TABLE table=default.Table_2f5270fd3d3e49eeab18022f7aa3f7a3 [id=b80483b9dffd4b1b9e75e2af41fc75e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.658813 1171 tablet_bootstrap.cc:654] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.662640 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 86e9154321d7423ba69478f94b60ee8c (DEFAULT_TABLE table=default.Table_b64d5e1e2ea54fbdbf2fd4d9cdea9fe4 [id=7081ea42230a4b4a8be0f4cdad83e043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.636134 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e2bc4df39674dc3be0cd32c68f1ebca" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:19.664245 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e700f0cef0e64bd8833bb9c09bb424f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.665728 883 tablet_service.cc:1468] Processing CreateTablet for tablet 7e8396ea40d34c66ab740c66647a6afc (DEFAULT_TABLE table=default.Table_2f5270fd3d3e49eeab18022f7aa3f7a3 [id=b80483b9dffd4b1b9e75e2af41fc75e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.669955 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1234) took 4538 ms. Trace:
I20250625 13:52:19.670886 886 tablet_service.cc:1468] Processing CreateTablet for tablet ae8af48a422c4a76b2e5440f1b75dfb3 (DEFAULT_TABLE table=default.Table_3ddbb86e81474e9ca79571a9c0372b74 [id=449814f10d87435e9e60bf524bc61f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.671263 913 raft_consensus.cc:3058] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.672039 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fdb54fda5d4458959fdea9102b8ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.672297 1170 tablet_bootstrap.cc:492] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:19.676671 753 tablet_service.cc:1468] Processing CreateTablet for tablet 227023f1cd3a49649cf5b3a08f604811 (DEFAULT_TABLE table=default.Table_deb3b74de40448608a132c85f19617d0 [id=56c7b77fb67b4a8e92765c904dfa831b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.676110 1170 ts_tablet_manager.cc:1397] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.002s
I20250625 13:52:19.679318 746 tablet_service.cc:1468] Processing CreateTablet for tablet c0f17bbc71994878a6e594657dc33fa9 (DEFAULT_TABLE table=default.Table_d15a9f7239f84b448320dfae951349cf [id=70ff0f1aa8614f5c894f0adc3f88bb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.679816 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae8af48a422c4a76b2e5440f1b75dfb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.681139 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e9154321d7423ba69478f94b60ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.681840 745 tablet_service.cc:1468] Processing CreateTablet for tablet 7e8396ea40d34c66ab740c66647a6afc (DEFAULT_TABLE table=default.Table_2f5270fd3d3e49eeab18022f7aa3f7a3 [id=b80483b9dffd4b1b9e75e2af41fc75e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.682852 892 tablet_service.cc:1468] Processing CreateTablet for tablet 86e9154321d7423ba69478f94b60ee8c (DEFAULT_TABLE table=default.Table_b64d5e1e2ea54fbdbf2fd4d9cdea9fe4 [id=7081ea42230a4b4a8be0f4cdad83e043]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:19.683053 849 consensus_peers.cc:487] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:19.682586 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ad205ee6e647fc87434079c444e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.684191 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fdb54fda5d4458959fdea9102b8ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.685372 1168 rpcz_store.cc:276] 0625 13:52:15.098236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.098305 (+ 69us) service_pool.cc:225] Handling call
0625 13:52:19.634191 (+4535886us) ts_tablet_manager.cc:1145] Deleting tablet 6e2d109e6fce407dad72a2469479b5ed
0625 13:52:19.636505 (+ 2314us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.636614 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535705}
I20250625 13:52:19.686065 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f3a9c1627246af8f020cd75480941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.686312 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc60ebdbced54d67b6cb712fc04ce85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.686501 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e700f0cef0e64bd8833bb9c09bb424f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.686725 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd5c72f4dd54e018848cac47fb1ec7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.688266 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ad205ee6e647fc87434079c444e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.688396 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 618413d2f9e549ae83e86c4eab280bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.688694 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f17bbc71994878a6e594657dc33fa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.690227 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1237) took 4481 ms. Trace:
I20250625 13:52:19.690326 1168 rpcz_store.cc:276] 0625 13:52:15.204023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.204089 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:19.685688 (+4481599us) ts_tablet_manager.cc:1145] Deleting tablet 17cf1172bcc74cca930824f9cda95f3b
0625 13:52:19.685699 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.685788 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4481439}
I20250625 13:52:19.690635 1168 tablet_replica.cc:331] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:19.690649 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e8396ea40d34c66ab740c66647a6afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.690827 1168 raft_consensus.cc:2241] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:19.691097 1168 pending_rounds.cc:70] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Trying to abort 1 pending ops.
I20250625 13:52:19.691156 1168 pending_rounds.cc:77] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520669670170624 op_type: NO_OP noop_request { }
I20250625 13:52:19.691818 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ad205ee6e647fc87434079c444e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.693325 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e1880a9b711f4ee19126d38f7bf88fe8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.693936 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1880a9b711f4ee19126d38f7bf88fe8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.691262 1168 raft_consensus.cc:2887] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:19.694181 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 7c7232b96d6c4827affe0f6ec30c0c88 (DEFAULT_TABLE table=default.Table_d336a3fe94264da6994165e69d9daa99 [id=8ef9d0c6b20c4c3a823bf8f9cf8c5180]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.697894 985 leader_election.cc:304] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:19.698071 747 tablet_service.cc:1468] Processing CreateTablet for tablet 35166d655a334fabb5f59fbebb1cb00f (DEFAULT_TABLE table=default.Table_9998378b240c46bcb121279c983960cc [id=48466b87e16942b59e1a05c811ecfd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.700893 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 227023f1cd3a49649cf5b3a08f604811. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.703075 2638 raft_consensus.cc:2747] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:19.703250 2638 raft_consensus.cc:491] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.703420 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35166d655a334fabb5f59fbebb1cb00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.703368 2638 raft_consensus.cc:513] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:19.703948 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae8af48a422c4a76b2e5440f1b75dfb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.704000 2638 leader_election.cc:290] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.707533 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58128254acf44129f7b4e452ac03fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.708828 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 70a2b1d20fed4de4a65f52689c397c6e (DEFAULT_TABLE table=default.Table_eec2951ee3b343b79d196d6f67178517 [id=87472718219e494cb145e7551b300dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.709586 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6fdb54fda5d4458959fdea9102b8ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.709811 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e52053e98544e8c9196ce1f3a0070c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.709867 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae8af48a422c4a76b2e5440f1b75dfb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.710165 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e9154321d7423ba69478f94b60ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.710250 1168 raft_consensus.cc:2270] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.712024 877 tablet_service.cc:1468] Processing CreateTablet for tablet 7c7232b96d6c4827affe0f6ec30c0c88 (DEFAULT_TABLE table=default.Table_d336a3fe94264da6994165e69d9daa99 [id=8ef9d0c6b20c4c3a823bf8f9cf8c5180]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.711036 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a2b1d20fed4de4a65f52689c397c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.715535 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e8396ea40d34c66ab740c66647a6afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.716107 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d76d69162374150ad261bbe4854b863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.718576 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e8396ea40d34c66ab740c66647a6afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.719848 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9e96f4e40f0c4f01b0f30c1502b3b7b4" } new_table_name: "default.Table_4f87ee4797da4b4484f89f91501523fa" modify_external_catalogs: true
I20250625 13:52:19.720192 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_968458660b3b479aa6fc0afcbd1d5925" } new_table_name: "default.Table_26108cd6ffa64c798bdefd834cf46129" modify_external_catalogs: true
I20250625 13:52:19.716051 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7d888e628a94f939ecf2571e60b1948" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.720402 916 raft_consensus.cc:2466] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:19.720502 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8744dd01ba5e4d3abbdfebd5e7b0bed4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:19.720885 983 leader_election.cc:304] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:19.721055 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8744dd01ba5e4d3abbdfebd5e7b0bed4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.721269 2638 raft_consensus.cc:2802] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:19.721339 2638 raft_consensus.cc:491] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:19.721398 2638 raft_consensus.cc:3058] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.721335 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_1fb6c2cf80f242568459c13aae98c60f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.721787 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fb6c2cf80f242568459c13aae98c60f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.724298 2638 raft_consensus.cc:513] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:19.724975 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27bb0aa3571b432b8bf5b8c70ef06f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.725250 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7232b96d6c4827affe0f6ec30c0c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.725694 2638 leader_election.cc:290] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.726100 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7d888e628a94f939ecf2571e60b1948" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:19.726305 916 raft_consensus.cc:3058] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.727138 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 93d0f3a82b574dd5bb952e9033d15e2e (DEFAULT_TABLE table=default.Table_2de49834c22b4922b4d0bdd92dd76e3f [id=60d58942d03b4b04be7f96c08f758a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.727553 2573 raft_consensus.cc:491] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.727679 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 628cb7cbe1324b9fb81f09bfb2fe1726 (DEFAULT_TABLE table=default.Table_3a318be2bb4949268060c27e670001ba [id=8ddbfd7a627d406384bc893c7173d988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.727725 2573 raft_consensus.cc:513] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.728168 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 8146355663ad48469756034e35a081ef (DEFAULT_TABLE table=default.Table_7ba3974781cc491bb035136be8634179 [id=d8a06910c1b2423aafb3677bbbaf4963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.728600 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 0220651a7528463dbb4257d06a737dd1 (DEFAULT_TABLE table=default.Table_2a8a7faddcd5494fa349ad6ecc6aec18 [id=3ee8fb3325074059aeed265c750e3d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.729056 1029 tablet_service.cc:1468] Processing CreateTablet for tablet d22d2b1cd6cb4c3b8e1bdc168304c6dc (DEFAULT_TABLE table=default.Table_56e5c59a581d431cada560e8606965ba [id=4aa0d48db54c44b68d36dd440cc11119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.729458 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7232b96d6c4827affe0f6ec30c0c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.729467 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed3c177bb6dd488aa54bee2a447016ff" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:19.729698 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 628cb7cbe1324b9fb81f09bfb2fe1726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.730011 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8146355663ad48469756034e35a081ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.730296 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0220651a7528463dbb4257d06a737dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.729507 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 3775c883f8534c6d90687152b9ca22d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.730571 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d0f3a82b574dd5bb952e9033d15e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.730808 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d22d2b1cd6cb4c3b8e1bdc168304c6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.731460 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a2e585b417224734b87815513bddfaa4 (DEFAULT_TABLE table=default.Table_ec46a4ca51884b13aa80806684369744 [id=0d9731930b4546c0b37dcac70722a0b8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:19.735102 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 29e223675c7842b383865a270b97ce58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29e223675c7842b383865a270b97ce58 already in progress: creating tablet
W20250625 13:52:19.735499 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 0bc136dc84de44b49abcdf4100a86137 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bc136dc84de44b49abcdf4100a86137 already in progress: creating tablet
W20250625 13:52:19.735675 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2914ce9b4c40a0bc00d0dd5ac72221 already in progress: creating tablet
W20250625 13:52:19.736145 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 6e2d109e6fce407dad72a2469479b5ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2d109e6fce407dad72a2469479b5ed already in progress: creating tablet
I20250625 13:52:19.730851 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2f6b50e30b4bfd979399ef4bb8a7d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:19.736328 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 17cf1172bcc74cca930824f9cda95f3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17cf1172bcc74cca930824f9cda95f3b already in progress: creating tablet
I20250625 13:52:19.724522 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0220651a7528463dbb4257d06a737dd1 (DEFAULT_TABLE table=default.Table_2a8a7faddcd5494fa349ad6ecc6aec18 [id=3ee8fb3325074059aeed265c750e3d94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.739538 1168 ts_tablet_manager.cc:1905] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.742017 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0220651a7528463dbb4257d06a737dd1 (DEFAULT_TABLE table=default.Table_2a8a7faddcd5494fa349ad6ecc6aec18 [id=3ee8fb3325074059aeed265c750e3d94]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:19.743144 849 leader_election.cc:343] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:19.728690 2573 leader_election.cc:290] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:19.743853 896 tablet_service.cc:1468] Processing CreateTablet for tablet 93d0f3a82b574dd5bb952e9033d15e2e (DEFAULT_TABLE table=default.Table_2de49834c22b4922b4d0bdd92dd76e3f [id=60d58942d03b4b04be7f96c08f758a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.745005 888 tablet_service.cc:1468] Processing CreateTablet for tablet a2e585b417224734b87815513bddfaa4 (DEFAULT_TABLE table=default.Table_ec46a4ca51884b13aa80806684369744 [id=0d9731930b4546c0b37dcac70722a0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.745160 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0220651a7528463dbb4257d06a737dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.745417 895 tablet_service.cc:1468] Processing CreateTablet for tablet 70a2b1d20fed4de4a65f52689c397c6e (DEFAULT_TABLE table=default.Table_eec2951ee3b343b79d196d6f67178517 [id=87472718219e494cb145e7551b300dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.756392 889 tablet_service.cc:1468] Processing CreateTablet for tablet d22d2b1cd6cb4c3b8e1bdc168304c6dc (DEFAULT_TABLE table=default.Table_56e5c59a581d431cada560e8606965ba [id=4aa0d48db54c44b68d36dd440cc11119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.755640 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce10088771c49f7844aa074d3d58a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.757344 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8146355663ad48469756034e35a081ef (DEFAULT_TABLE table=default.Table_7ba3974781cc491bb035136be8634179 [id=d8a06910c1b2423aafb3677bbbaf4963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.759246 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e585b417224734b87815513bddfaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.768352 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a2b1d20fed4de4a65f52689c397c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.768683 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 3775c883f8534c6d90687152b9ca22d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.771106 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59e6239a59d4f188ee768fe1c2b4df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.771519 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0220651a7528463dbb4257d06a737dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.771868 1165 ts_tablet_manager.cc:1918] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:19.771991 1165 log.cc:1199] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/4ae5f1d414794474b2178b431cbc1d44
I20250625 13:52:19.773185 882 tablet_service.cc:1468] Processing CreateTablet for tablet 628cb7cbe1324b9fb81f09bfb2fe1726 (DEFAULT_TABLE table=default.Table_3a318be2bb4949268060c27e670001ba [id=8ddbfd7a627d406384bc893c7173d988]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.777814 1170 raft_consensus.cc:357] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.787187 1170 raft_consensus.cc:383] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.789974 1170 raft_consensus.cc:738] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.793581 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e585b417224734b87815513bddfaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.795235 1165 ts_tablet_manager.cc:1939] T 4ae5f1d414794474b2178b431cbc1d44 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:19.795504 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 628cb7cbe1324b9fb81f09bfb2fe1726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.799142 1168 ts_tablet_manager.cc:1918] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:19.803424 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c95698028b9241c880fe126c21813367" } modify_external_catalogs: true
I20250625 13:52:19.808246 2638 raft_consensus.cc:491] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.808427 2638 raft_consensus.cc:513] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.809123 2638 leader_election.cc:290] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.811300 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_ee5d5978aaf540b8bc77e7c4bbe55258" } modify_external_catalogs: true
I20250625 13:52:19.813298 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3f5dc7a3e577405391f6d3ec22c932ff" } modify_external_catalogs: true
I20250625 13:52:19.817374 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 0cf35321b38a4e81b41563f711dbb86d
I20250625 13:52:19.818153 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cf35321b38a4e81b41563f711dbb86d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.819811 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 34fe02d7b37f4d99bbae082943223e35
I20250625 13:52:19.821132 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 34fe02d7b37f4d99bbae082943223e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.821259 1170 consensus_queue.cc:260] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.821538 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_f0eaf57415454faa8ab987a09dbe77cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:19.821808 1170 ts_tablet_manager.cc:1428] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.143s user 0.002s sys 0.000s
W20250625 13:52:19.822075 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0eaf57415454faa8ab987a09dbe77cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.822098 1170 tablet_bootstrap.cc:492] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:19.823130 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 835e221fb6fa4e8b83a15fda0f1951af consensus state has no leader
I20250625 13:52:19.800503 2254 ts_tablet_manager.cc:1918] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:19.823278 2254 log.cc:1199] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/8063122ea7b24b6790f5c3cc884e09be
I20250625 13:52:19.823624 2254 ts_tablet_manager.cc:1939] T 8063122ea7b24b6790f5c3cc884e09be P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:19.824065 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f44d0f54bc24feb8386cfce1cd67db6 consensus state has no leader
I20250625 13:52:19.824388 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_92e864b5eb8e425bb39f36d08dcf5859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.824864 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92e864b5eb8e425bb39f36d08dcf5859 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.825682 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_c5a0db24bad64fdf840914eda05b9467"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.826452 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5a0db24bad64fdf840914eda05b9467 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.826855 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a4182ebe832a4ef2a6b0a736bfcbc4b4" } new_table_name: "default.Table_316ec585c83942ba9400d9b992279d02" modify_external_catalogs: true
I20250625 13:52:19.829586 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 50ac647b1d704142b94da8d506319466
I20250625 13:52:19.831059 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet ba762a8edb8847368103e5bede9e9d0b
I20250625 13:52:19.831707 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet ba762a8edb8847368103e5bede9e9d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.832535 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_854c6918e04c4dc69a3b9377139e7b80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.833309 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_854c6918e04c4dc69a3b9377139e7b80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:19.834857 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet ef2b08fba37f4a38880c018ec2d869c8 (table default.Table_3a4e5f10c2ec453a82fb97800a0d37f7 [id=8b35aa4ce9864529b6af2d4aeb6d9eff]). Expected version 2 got 0
I20250625 13:52:19.845666 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 50ac647b1d704142b94da8d506319466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.802850 913 raft_consensus.cc:2466] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:19.810746 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a2007e51d424053bc2836dedf46add2" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.846249 912 raft_consensus.cc:2466] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:19.810796 916 raft_consensus.cc:2466] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:19.810900 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d22d2b1cd6cb4c3b8e1bdc168304c6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.846019 1168 log.cc:1199] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/63d01af6b320489fb99c467c11ed1dc0
I20250625 13:52:19.847609 1168 ts_tablet_manager.cc:1939] T 63d01af6b320489fb99c467c11ed1dc0 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:19.846655 983 leader_election.cc:304] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:19.848031 983 leader_election.cc:304] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:19.848284 983 leader_election.cc:304] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:19.848606 2636 raft_consensus.cc:2802] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:19.851846 2637 raft_consensus.cc:2802] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:19.851959 2637 raft_consensus.cc:491] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:19.852047 2637 raft_consensus.cc:3058] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.857400 2637 raft_consensus.cc:513] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.876266 2637 leader_election.cc:290] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.876451 2637 raft_consensus.cc:2802] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:19.876552 2637 raft_consensus.cc:695] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:19.876771 2637 consensus_queue.cc:237] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:19.884142 742 tablet_service.cc:1468] Processing CreateTablet for tablet 7c7232b96d6c4827affe0f6ec30c0c88 (DEFAULT_TABLE table=default.Table_d336a3fe94264da6994165e69d9daa99 [id=8ef9d0c6b20c4c3a823bf8f9cf8c5180]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.887454 752 tablet_service.cc:1468] Processing CreateTablet for tablet 70a2b1d20fed4de4a65f52689c397c6e (DEFAULT_TABLE table=default.Table_eec2951ee3b343b79d196d6f67178517 [id=87472718219e494cb145e7551b300dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.888178 2594 raft_consensus.cc:491] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.889223 747 tablet_service.cc:1468] Processing CreateTablet for tablet 86e9154321d7423ba69478f94b60ee8c (DEFAULT_TABLE table=default.Table_b64d5e1e2ea54fbdbf2fd4d9cdea9fe4 [id=7081ea42230a4b4a8be0f4cdad83e043]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:19.890904 1166 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.009s sys 0.006s
W20250625 13:52:19.891058 1166 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.518s user 0.009s sys 0.006s
I20250625 13:52:19.893715 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7232b96d6c4827affe0f6ec30c0c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.835708 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_9d0e64e523c24f2ca751d55bb49a7213" } new_table_name: "default.Table_6a976a07423a4a38aac2b4fe974cfd83" modify_external_catalogs: true
I20250625 13:52:19.835747 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_afdcf859399e4102b6836c6ffe61ef4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:19.848326 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_437ef29da8a64a32a591093ac536eadc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:19.898617 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afdcf859399e4102b6836c6ffe61ef4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:19.898978 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_437ef29da8a64a32a591093ac536eadc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:19.851521 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 189a9347cd5d4b5f955e5c0706aa3947 consensus state has no leader
I20250625 13:52:19.854130 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_e40aed94ebc340d8b6324a1bd2840e03" } new_table_name: "default.Table_888cac7eddc64cc2b5d63aee3415b9c0" modify_external_catalogs: true
I20250625 13:52:19.855736 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_add52e39ed4c414c9c9e930208ab5c10" } modify_external_catalogs: true
I20250625 13:52:19.900861 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_add52e39ed4c414c9c9e930208ab5c10" } modify_external_catalogs: true
I20250625 13:52:19.905560 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (table default.Table_727a4473adab487fa65f094237ec794b [id=3977ee3968d246b7a8aed1fafd1822b7]). Expected version 3 got 0
I20250625 13:52:19.908119 2656 raft_consensus.cc:491] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.908331 2656 raft_consensus.cc:513] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.909272 2656 leader_election.cc:290] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.910588 2636 raft_consensus.cc:695] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:19.910971 2636 consensus_queue.cc:237] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:19.912591 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:34247, user_credentials={real_user=slave}} blocked reactor thread for 64313.9us
W20250625 13:52:19.914691 985 leader_election.cc:343] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9a2007e51d424053bc2836dedf46add2
I20250625 13:52:19.914443 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed3c177bb6dd488aa54bee2a447016ff" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:19.916918 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1350) took 1291 ms. Trace:
I20250625 13:52:19.917058 1165 rpcz_store.cc:276] 0625 13:52:18.625498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.625576 (+ 78us) service_pool.cc:225] Handling call
0625 13:52:19.436809 (+811233us) ts_tablet_manager.cc:1145] Deleting tablet 4ae5f1d414794474b2178b431cbc1d44
0625 13:52:19.436820 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.452539 (+ 15719us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.771731 (+319192us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.916893 (+145162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":811019}
I20250625 13:52:19.916028 2662 tablet.cc:1722] T d7c6a325d40c4f7ba2f342e39b9437e3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250625 13:52:19.926592 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2456662afcc3426c8dbc5e1132c2f3ac consensus state has no leader
I20250625 13:52:19.926625 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 50786a4c25274ffb8f9910c5dafe3933
W20250625 13:52:19.929015 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce10088771c49f7844aa074d3d58a6c consensus state has no leader
W20250625 13:52:19.931111 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:39291, user_credentials={real_user=slave}} blocked reactor thread for 106581us
I20250625 13:52:19.840734 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cf35321b38a4e81b41563f711dbb86d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.842564 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 50ac647b1d704142b94da8d506319466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.842684 880 tablet_service.cc:1515] Processing DeleteTablet for tablet ba762a8edb8847368103e5bede9e9d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.844537 1171 tablet_bootstrap.cc:492] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:19.855518 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d0f3a82b574dd5bb952e9033d15e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.939724 1171 ts_tablet_manager.cc:1397] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.507s user 0.005s sys 0.001s
I20250625 13:52:19.940496 1171 raft_consensus.cc:357] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.940672 1171 raft_consensus.cc:383] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:19.940716 1171 raft_consensus.cc:738] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:19.940919 1171 consensus_queue.cc:260] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.941428 1171 ts_tablet_manager.cc:1428] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:19.942004 1171 tablet_bootstrap.cc:492] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:19.943328 1171 tablet_bootstrap.cc:654] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:19.943684 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8146355663ad48469756034e35a081ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.836407 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 34fe02d7b37f4d99bbae082943223e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.908293 889 tablet_service.cc:1515] Processing DeleteTablet for tablet afa1c198dd6541809843244c7cc18194 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.913257 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a2007e51d424053bc2836dedf46add2" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:19.946547 915 raft_consensus.cc:3058] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.949142 915 raft_consensus.cc:2466] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:19.913543 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077e8c001cc646559b5268d4a48c0ad6" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:19.949525 914 raft_consensus.cc:2466] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:19.918394 2642 tablet.cc:1722] T d7c6a325d40c4f7ba2f342e39b9437e3 P affa1c3035b940e88ca143be7d35840d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:19.949777 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 50786a4c25274ffb8f9910c5dafe3933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:19.950028 983 leader_election.cc:304] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:19.950279 983 leader_election.cc:304] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:19.951296 768 raft_consensus.cc:2466] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:19.951951 985 leader_election.cc:343] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 077e8c001cc646559b5268d4a48c0ad6
I20250625 13:52:19.952749 2660 raft_consensus.cc:2802] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:19.952873 2660 raft_consensus.cc:695] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:19.953114 2660 consensus_queue.cc:237] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.954159 2638 raft_consensus.cc:2802] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:19.954233 2638 raft_consensus.cc:491] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:19.954285 2638 raft_consensus.cc:3058] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.954531 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1239) took 4725 ms. Trace:
I20250625 13:52:19.954599 1168 rpcz_store.cc:276] 0625 13:52:15.229139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.229193 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:19.690586 (+4461393us) ts_tablet_manager.cc:1145] Deleting tablet 63d01af6b320489fb99c467c11ed1dc0
0625 13:52:19.690596 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.782505 (+ 91909us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.799054 (+ 16549us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.954515 (+155461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4461128}
I20250625 13:52:19.954911 1168 tablet_replica.cc:331] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:19.955065 1168 raft_consensus.cc:2241] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.955184 1168 raft_consensus.cc:2270] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.956496 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 63d01af6b320489fb99c467c11ed1dc0 (table default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae [id=c4babc967a48425587250196c8d9b4ab]) successfully deleted
I20250625 13:52:19.957234 2638 raft_consensus.cc:513] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.958431 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a2b1d20fed4de4a65f52689c397c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.958635 754 tablet_service.cc:1468] Processing CreateTablet for tablet 93d0f3a82b574dd5bb952e9033d15e2e (DEFAULT_TABLE table=default.Table_2de49834c22b4922b4d0bdd92dd76e3f [id=60d58942d03b4b04be7f96c08f758a83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.958961 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d0f3a82b574dd5bb952e9033d15e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.959846 753 tablet_service.cc:1468] Processing CreateTablet for tablet a2e585b417224734b87815513bddfaa4 (DEFAULT_TABLE table=default.Table_ec46a4ca51884b13aa80806684369744 [id=0d9731930b4546c0b37dcac70722a0b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.960085 2638 leader_election.cc:290] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:19.960264 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e585b417224734b87815513bddfaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:19.960377 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1351) took 1322 ms. Trace:
I20250625 13:52:19.960515 1165 rpcz_store.cc:276] 0625 13:52:18.638353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.638419 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:19.917490 (+1279071us) ts_tablet_manager.cc:1145] Deleting tablet 2191b836bb3044c89fee13c869d9e07d
0625 13:52:19.960238 (+ 42748us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.960362 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1277088}
I20250625 13:52:19.960763 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 7506cce0ec4c4ec68f25591cb98b1252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.960892 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1352) took 1322 ms. Trace:
I20250625 13:52:19.960958 1165 rpcz_store.cc:276] 0625 13:52:18.638471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.638595 (+ 124us) service_pool.cc:225] Handling call
0625 13:52:19.960798 (+1322203us) ts_tablet_manager.cc:1145] Deleting tablet 391b412dbb9e420c90aa79473bbbe8d7
0625 13:52:19.960806 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.960877 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320338}
I20250625 13:52:19.961263 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1354) took 1322 ms. Trace:
I20250625 13:52:19.961755 741 tablet_service.cc:1468] Processing CreateTablet for tablet d22d2b1cd6cb4c3b8e1bdc168304c6dc (DEFAULT_TABLE table=default.Table_56e5c59a581d431cada560e8606965ba [id=4aa0d48db54c44b68d36dd440cc11119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.963694 743 tablet_service.cc:1468] Processing CreateTablet for tablet 8146355663ad48469756034e35a081ef (DEFAULT_TABLE table=default.Table_7ba3974781cc491bb035136be8634179 [id=d8a06910c1b2423aafb3677bbbaf4963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:52:19.964131 744 tablet_service.cc:1468] Processing CreateTablet for tablet 628cb7cbe1324b9fb81f09bfb2fe1726 (DEFAULT_TABLE table=default.Table_3a318be2bb4949268060c27e670001ba [id=8ddbfd7a627d406384bc893c7173d988]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 13:52:19.964437 985 consensus_peers.cc:487] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a30a35e3639a43c5b078169060824723. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:19.964596 985 leader_election.cc:343] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 077e8c001cc646559b5268d4a48c0ad6
I20250625 13:52:19.930014 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 50786a4c25274ffb8f9910c5dafe3933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.966742 2594 raft_consensus.cc:513] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.968246 1166 tablet_bootstrap.cc:492] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:19.968412 1166 ts_tablet_manager.cc:1397] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.632s user 0.011s sys 0.007s
I20250625 13:52:19.974573 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077e8c001cc646559b5268d4a48c0ad6" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:19.974790 914 raft_consensus.cc:3058] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:19.975468 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1316) took 2027 ms. Trace:
I20250625 13:52:19.975631 2254 rpcz_store.cc:276] 0625 13:52:17.948185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.948248 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:19.337884 (+1389636us) ts_tablet_manager.cc:1145] Deleting tablet 8063122ea7b24b6790f5c3cc884e09be
0625 13:52:19.337896 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.614728 (+276832us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.800384 (+185656us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:19.975449 (+175065us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1389363}
I20250625 13:52:19.976054 2254 tablet_replica.cc:331] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:19.976269 2254 raft_consensus.cc:2241] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:19.976476 2254 raft_consensus.cc:2270] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:19.977221 2254 ts_tablet_manager.cc:1905] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:19.977339 914 raft_consensus.cc:2466] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:19.977481 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 8063122ea7b24b6790f5c3cc884e09be (table default.Table_1ffafeb50d654290963bf41dfeb57f55 [id=75f0bde5d60641f8afe279bd82d16ae1]) successfully deleted
I20250625 13:52:19.978554 590 catalog_manager.cc:5674] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247) does not have the latest schema for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (table default.Table_727a4473adab487fa65f094237ec794b [id=3977ee3968d246b7a8aed1fafd1822b7]). Expected version 3 got 0
I20250625 13:52:19.982697 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ca3f3760e6e41a29d004e6afe63ec98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:19.961325 1165 rpcz_store.cc:276] 0625 13:52:18.638662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.638708 (+ 46us) service_pool.cc:225] Handling call
0625 13:52:19.961166 (+1322458us) ts_tablet_manager.cc:1145] Deleting tablet 3fc03dbe0a19470fbd0fe550159cd232
0625 13:52:19.961176 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:19.961249 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320501}
I20250625 13:52:19.981284 983 leader_election.cc:304] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:19.992921 2636 raft_consensus.cc:2802] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:19.993201 2636 raft_consensus.cc:695] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:19.993506 2636 consensus_queue.cc:237] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:19.994127 985 leader_election.cc:343] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b7d888e628a94f939ecf2571e60b1948
I20250625 13:52:19.994979 2573 raft_consensus.cc:491] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:19.995167 2573 raft_consensus.cc:513] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:19.995947 2573 leader_election.cc:290] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:19.997468 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2de9656958164dc4ae2fb0078b38f6c3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:19.998322 849 leader_election.cc:343] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.002563 1171 tablet_bootstrap.cc:492] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.003218 2594 leader_election.cc:290] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.002704 1171 ts_tablet_manager.cc:1397] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.003s
I20250625 13:52:20.003827 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69c44d72516e4883aff636ca65929c11" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:20.003917 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69c44d72516e4883aff636ca65929c11" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.004252 1171 raft_consensus.cc:357] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.004401 1171 raft_consensus.cc:383] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.004446 1171 raft_consensus.cc:738] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:52:20.004385 711 leader_election.cc:343] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:20.004611 711 leader_election.cc:343] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.004614 1171 consensus_queue.cc:260] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.004701 711 leader_election.cc:304] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:20.005065 1171 ts_tablet_manager.cc:1428] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:20.005252 2594 raft_consensus.cc:2747] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.007051 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 4ae5f1d414794474b2178b431cbc1d44 (table default.Table_467fcbf2d0ba44108a7aa82294efb342 [id=5b4f30f7fe414c589499632bdd0f8922]) successfully deleted
W20250625 13:52:20.007519 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 2191b836bb3044c89fee13c869d9e07d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2191b836bb3044c89fee13c869d9e07d already in progress: creating tablet
W20250625 13:52:20.007740 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 391b412dbb9e420c90aa79473bbbe8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 391b412dbb9e420c90aa79473bbbe8d7 already in progress: creating tablet
W20250625 13:52:20.008141 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 3fc03dbe0a19470fbd0fe550159cd232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fc03dbe0a19470fbd0fe550159cd232 already in progress: creating tablet
I20250625 13:52:20.008896 1170 tablet_bootstrap.cc:654] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.008992 851 leader_election.cc:304] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:20.009727 2631 consensus_queue.cc:1035] T df72a1a461aa44fe80cff6e3870da8e7 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:20.009863 1168 ts_tablet_manager.cc:1905] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.010141 2631 raft_consensus.cc:2802] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:20.010212 2631 raft_consensus.cc:491] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:20.010278 2631 raft_consensus.cc:3058] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.012894 2631 raft_consensus.cc:513] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.013553 2631 leader_election.cc:290] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:20.016134 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed3c177bb6dd488aa54bee2a447016ff" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:20.016959 849 leader_election.cc:343] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.024631 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1c3914808b7547c7851ed153bbbcddb5" } modify_external_catalogs: true
I20250625 13:52:20.024830 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1c3914808b7547c7851ed153bbbcddb5" } modify_external_catalogs: true
I20250625 13:52:20.025574 590 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: default.Table_727a4473adab487fa65f094237ec794b [id=3977ee3968d246b7a8aed1fafd1822b7] alter complete (version 3)
I20250625 13:52:20.026211 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_12e1df9c1f1c49e8bc2a34c01de01d13" } new_table_name: "default.Table_bb6eed6d946d4e7097bf4665b269c72a" modify_external_catalogs: true
I20250625 13:52:20.027539 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_cebdb854140148d38efad7064c0a9034"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.028033 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cebdb854140148d38efad7064c0a9034 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.030640 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d373cb07af7a453dad8f774642a336dd" } new_table_name: "default.Table_6b731068babc4283b031e20d7159b305" modify_external_catalogs: true
I20250625 13:52:20.031944 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8146355663ad48469756034e35a081ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:20.032805 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet aa555aa6a0124e5bac31c2bb83bb26ae
I20250625 13:52:20.033622 882 tablet_service.cc:1515] Processing DeleteTablet for tablet aa555aa6a0124e5bac31c2bb83bb26ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:20.034214 985 leader_election.cc:343] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: b7d888e628a94f939ecf2571e60b1948
I20250625 13:52:20.035406 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 78c79a2f6544493ba3f3ec20ea67fa95 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:20.035466 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f6b4ad595d094974adb62da4fe761a46" } modify_external_catalogs: true
I20250625 13:52:20.035583 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f6b4ad595d094974adb62da4fe761a46" } modify_external_catalogs: true
I20250625 13:52:20.037825 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_06c39e7e718042edb9d263fd11f714f0" } modify_external_catalogs: true
I20250625 13:52:20.037937 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_06c39e7e718042edb9d263fd11f714f0" } modify_external_catalogs: true
I20250625 13:52:20.038698 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e9154321d7423ba69478f94b60ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:52:20.038704 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_28d906d15b174e3aad20a5e224613fe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:20.039001 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 628cb7cbe1324b9fb81f09bfb2fe1726. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:20.039161 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28d906d15b174e3aad20a5e224613fe1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.040427 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_3f7343f2a4884c55b9c63dfadb28b025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:20.040650 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c75deb64c74f0282efeed827659ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:12 UTC) from {username='slave'} at 127.0.0.1:42238
W20250625 13:52:20.040913 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f7343f2a4884c55b9c63dfadb28b025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.045415 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d22d2b1cd6cb4c3b8e1bdc168304c6dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 13:52:20.046444 985 leader_election.cc:343] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9a2007e51d424053bc2836dedf46add2
I20250625 13:52:20.046821 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_91836dc58519406faa07be52d36ed947" } modify_external_catalogs: true
I20250625 13:52:20.046960 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_91836dc58519406faa07be52d36ed947" } modify_external_catalogs: true
I20250625 13:52:20.049172 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6c350d8935c74074855cb7a8c48e9e17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.049681 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c350d8935c74074855cb7a8c48e9e17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:20.049853 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93a0b94879874446b569b3cb3250b185 consensus state has no leader
I20250625 13:52:20.050305 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 78c79a2f6544493ba3f3ec20ea67fa95
I20250625 13:52:20.050721 591 catalog_manager.cc:6652] ReplaceTablet: tablet 78c79a2f6544493ba3f3ec20ea67fa95 deleted and replaced by tablet 7b08c46e0c7a4a67976c6a75460fa435
I20250625 13:52:20.051123 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d12f07f4145240958d7decf21c03b6bc
I20250625 13:52:20.052700 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 78c79a2f6544493ba3f3ec20ea67fa95 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b08c46e0c7a4a67976c6a75460fa435) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:20.053596 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 5af2b974df3a41f7b60537c9d8eac1e6
I20250625 13:52:20.054340 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 5af2b974df3a41f7b60537c9d8eac1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:20.054858 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9a5bcf4a7024ee39349358b15c7379b consensus state has no leader
I20250625 13:52:20.054872 1008 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:42232 (request call id 188) took 1130 ms. Trace:
I20250625 13:52:20.054972 1008 rpcz_store.cc:276] 0625 13:52:18.924150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.924209 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:20.054834 (+1130625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:52:20.056073 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8a9e2586f6d0486faafe1b8311f52962"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.056608 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a9e2586f6d0486faafe1b8311f52962 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.058184 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_9918069c31e945819ad208720537e6ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.058657 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9918069c31e945819ad208720537e6ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.060699 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7eb6a8becb2843348ce594ffa2ee291c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:20.060827 1168 ts_tablet_manager.cc:1918] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:20.060952 1168 log.cc:1199] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/cbe4046742b34f1ab0b1610b5d6ee01e
W20250625 13:52:20.061219 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7eb6a8becb2843348ce594ffa2ee291c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.061302 1168 ts_tablet_manager.cc:1939] T cbe4046742b34f1ab0b1610b5d6ee01e P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:20.064497 879 tablet_service.cc:1515] Processing DeleteTablet for tablet d12f07f4145240958d7decf21c03b6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:20.065210 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 69c44d72516e4883aff636ca65929c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.065397 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3775c883f8534c6d90687152b9ca22d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.039669 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9efc7631df4ad29112a41f229ae2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:59 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.071748 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 34fe02d7b37f4d99bbae082943223e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.076257 878 tablet_service.cc:1515] Processing DeleteTablet for tablet d9462da7574a46839763b1c388102e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:20.078608 590 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f7d7b1e3d02142328521c5d3e2f25a9d
I20250625 13:52:20.086030 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 50ac647b1d704142b94da8d506319466 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.086254 757 tablet_service.cc:1515] Processing DeleteTablet for tablet ba762a8edb8847368103e5bede9e9d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.086366 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 50786a4c25274ffb8f9910c5dafe3933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.088204 757 tablet_service.cc:1515] Processing DeleteTablet for tablet aa555aa6a0124e5bac31c2bb83bb26ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.088824 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 78c79a2f6544493ba3f3ec20ea67fa95 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b08c46e0c7a4a67976c6a75460fa435) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.089047 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet aa555aa6a0124e5bac31c2bb83bb26ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.091189 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet d12f07f4145240958d7decf21c03b6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.091396 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 5af2b974df3a41f7b60537c9d8eac1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.091636 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 78c79a2f6544493ba3f3ec20ea67fa95 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b08c46e0c7a4a67976c6a75460fa435) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.092922 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1353) took 1454 ms. Trace:
I20250625 13:52:20.098423 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1240) took 4863 ms. Trace:
I20250625 13:52:20.099037 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ddde325144324d1f88fd7d68629a7323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.099639 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddde325144324d1f88fd7d68629a7323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.102160 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_bd28610323b44b249068082bd830bd60" } new_table_name: "default.Table_734ffa7636304a8b9eeaf6997d1292cd" modify_external_catalogs: true
I20250625 13:52:20.112748 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d5c89472636b4d788a3a059b209f2361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.113304 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5c89472636b4d788a3a059b209f2361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:20.116050 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 250b0ac801e842cbb39912f80f5e9b82 consensus state has no leader
I20250625 13:52:20.100402 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 5af2b974df3a41f7b60537c9d8eac1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.116469 1165 rpcz_store.cc:276] 0625 13:52:18.638560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.638713 (+ 153us) service_pool.cc:225] Handling call
0625 13:52:19.988793 (+1350080us) ts_tablet_manager.cc:1145] Deleting tablet 45079deff84a41b488ccdde2f5059431
0625 13:52:20.092659 (+103866us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.092895 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348079}
I20250625 13:52:20.129175 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_f3b3e3e07eb84bfa8a9396b3161a9114" } new_table_name: "default.Table_c9edb7436658409d87ac5b8d905e155e" modify_external_catalogs: true
I20250625 13:52:20.105819 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cf35321b38a4e81b41563f711dbb86d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:19 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.105935 757 tablet_service.cc:1515] Processing DeleteTablet for tablet d12f07f4145240958d7decf21c03b6bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:20.131253 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 45079deff84a41b488ccdde2f5059431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45079deff84a41b488ccdde2f5059431 already in progress: creating tablet
I20250625 13:52:20.114430 1166 raft_consensus.cc:357] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.131932 1166 raft_consensus.cc:383] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.132081 1166 raft_consensus.cc:738] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:20.096014 778 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.004s sys 0.025s
I20250625 13:52:20.133242 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_98ed87f3ecee4add88ff785fe7175ce1" } modify_external_catalogs: true
I20250625 13:52:20.133406 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_98ed87f3ecee4add88ff785fe7175ce1" } modify_external_catalogs: true
I20250625 13:52:20.133481 1168 rpcz_store.cc:276] 0625 13:52:15.235426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.235489 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:19.954870 (+4719381us) ts_tablet_manager.cc:1145] Deleting tablet cbe4046742b34f1ab0b1610b5d6ee01e
0625 13:52:19.954879 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.030075 (+ 75196us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.033142 (+ 3067us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.098396 (+ 65254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4719148}
I20250625 13:52:20.134007 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet cbe4046742b34f1ab0b1610b5d6ee01e (table default.Table_906b9e1a3a8147fb935f7a96968ed294 [id=317ce1c5a5774d6ca8d00ed976bf8615]) successfully deleted
I20250625 13:52:20.134742 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_a55e69e3ed874f37bfe161cf46b4dfb5" } new_table_name: "default.Table_c979e1bc235d46debf85733939967c4d" modify_external_catalogs: true
I20250625 13:52:20.137245 2204 raft_consensus.cc:491] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.137396 2204 raft_consensus.cc:513] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.138012 2204 leader_election.cc:290] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:20.139133 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ca3f3760e6e41a29d004e6afe63ec98" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.132586 778 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s user 0.004s sys 0.025s
W20250625 13:52:20.139794 849 leader_election.cc:343] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.132607 1166 consensus_queue.cc:260] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:20.142056 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09733b6641dd49c9b29633af1c466aa4 consensus state has no leader
I20250625 13:52:20.142446 1166 ts_tablet_manager.cc:1428] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.174s user 0.001s sys 0.001s
I20250625 13:52:20.108224 1171 tablet_bootstrap.cc:492] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
W20250625 13:52:20.144376 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5a50381bcaa4c51b6f6f5aa47237115 consensus state has no leader
I20250625 13:52:20.121294 2573 raft_consensus.cc:491] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.144615 2573 raft_consensus.cc:513] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.145283 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet d406206ea8e14121bf43214adfc5afda
I20250625 13:52:20.145370 2573 leader_election.cc:290] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:20.146180 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9462da7574a46839763b1c388102e60" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:20.146593 878 tablet_service.cc:1515] Processing DeleteTablet for tablet d406206ea8e14121bf43214adfc5afda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:52994
W20250625 13:52:20.146751 849 leader_election.cc:343] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.148254 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d406206ea8e14121bf43214adfc5afda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.149820 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 63d01af6b320489fb99c467c11ed1dc0 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:20.150197 1166 tablet_bootstrap.cc:492] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:20.154243 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1241) took 4890 ms. Trace:
I20250625 13:52:20.161798 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d406206ea8e14121bf43214adfc5afda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.166150 1168 rpcz_store.cc:276] 0625 13:52:15.263786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.263842 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:20.154109 (+4890267us) ts_tablet_manager.cc:1145] Deleting tablet f47980e0c9ad487b8113f13458e175ec
0625 13:52:20.154121 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.154221 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4890052}
I20250625 13:52:20.168632 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_797233f21b824f66ac77957bdc8230f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.169214 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_797233f21b824f66ac77957bdc8230f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:20.172786 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet f47980e0c9ad487b8113f13458e175ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47980e0c9ad487b8113f13458e175ec already in progress: creating tablet
I20250625 13:52:20.179401 2594 raft_consensus.cc:491] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.179610 2594 raft_consensus.cc:513] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.180408 2594 leader_election.cc:290] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.181134 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1242) took 4867 ms. Trace:
I20250625 13:52:20.183964 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9efc7631df4ad29112a41f229ae2e9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.185745 1168 rpcz_store.cc:276] 0625 13:52:15.313859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.313938 (+ 79us) service_pool.cc:225] Handling call
0625 13:52:20.180883 (+4866945us) ts_tablet_manager.cc:1145] Deleting tablet 9252dde349d24a8c99dfe2eda37c9f64
0625 13:52:20.180901 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.181106 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4866667}
W20250625 13:52:20.186158 711 leader_election.cc:343] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:20.186251 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 9252dde349d24a8c99dfe2eda37c9f64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9252dde349d24a8c99dfe2eda37c9f64 already in progress: creating tablet
I20250625 13:52:20.186829 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9efc7631df4ad29112a41f229ae2e9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:20.157752 1171 tablet_bootstrap.cc:654] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:20.187580 711 leader_election.cc:343] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:20.188885 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:44801, user_credentials={real_user=slave}} blocked reactor thread for 179299us
I20250625 13:52:20.189363 1170 tablet_bootstrap.cc:492] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.189427 2573 consensus_queue.cc:1035] T 669de0b7c93a4efea8623dab2e462c8d P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:20.189661 1170 ts_tablet_manager.cc:1397] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.368s user 0.005s sys 0.017s
W20250625 13:52:20.190483 851 leader_election.cc:343] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2de9656958164dc4ae2fb0078b38f6c3
I20250625 13:52:20.190675 851 leader_election.cc:304] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.190938 2631 raft_consensus.cc:2747] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.191020 1170 raft_consensus.cc:357] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.191232 1170 raft_consensus.cc:383] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.191347 1170 raft_consensus.cc:738] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.191618 1170 consensus_queue.cc:260] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.192147 1170 ts_tablet_manager.cc:1428] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:20.192502 1170 tablet_bootstrap.cc:492] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.192554 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed3c177bb6dd488aa54bee2a447016ff" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:20.192727 764 raft_consensus.cc:3058] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.212298 2631 raft_consensus.cc:491] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.212594 2631 raft_consensus.cc:513] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.187723 711 leader_election.cc:304] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:20.213843 2631 leader_election.cc:290] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:20.214111 2594 raft_consensus.cc:2747] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.214404 2631 raft_consensus.cc:491] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.214596 2631 raft_consensus.cc:513] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.214601 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da2f8d0d9a714a28ba6fb9642d461de2" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.215521 849 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.216239 2631 leader_election.cc:290] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:20.218801 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1243) took 4835 ms. Trace:
I20250625 13:52:20.219322 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7506cce0ec4c4ec68f25591cb98b1252" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.219956 849 leader_election.cc:343] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.220304 1168 rpcz_store.cc:276] 0625 13:52:15.383802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.383862 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:20.185997 (+4802135us) ts_tablet_manager.cc:1145] Deleting tablet d0baddb123f0401a98662b4b483b9587
0625 13:52:20.186006 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.186099 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4801952}
W20250625 13:52:20.222738 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d0baddb123f0401a98662b4b483b9587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0baddb123f0401a98662b4b483b9587 already in progress: creating tablet
I20250625 13:52:20.223256 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1257) took 4505 ms. Trace:
I20250625 13:52:20.225510 1168 rpcz_store.cc:276] 0625 13:52:15.717641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.717695 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:20.223136 (+4505441us) ts_tablet_manager.cc:1145] Deleting tablet 1696cbd156f94d14a5f1da76c8e58584
0625 13:52:20.223147 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.223240 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4505273}
I20250625 13:52:20.226044 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1369) took 1426 ms. Trace:
I20250625 13:52:20.226512 1165 rpcz_store.cc:276] 0625 13:52:18.799526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.799577 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:20.191372 (+1391795us) ts_tablet_manager.cc:1145] Deleting tablet aafb7eef813f48fc8295fbb30a29cd0d
0625 13:52:20.225710 (+ 34338us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.226022 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1349511}
I20250625 13:52:20.227105 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1371) took 1409 ms. Trace:
I20250625 13:52:20.229290 2594 raft_consensus.cc:491] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.229405 2594 raft_consensus.cc:513] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.230024 2594 leader_election.cc:290] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.231002 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.231698 1170 tablet_bootstrap.cc:654] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.231763 1171 tablet_bootstrap.cc:492] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
W20250625 13:52:20.231696 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 1696cbd156f94d14a5f1da76c8e58584 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1696cbd156f94d14a5f1da76c8e58584 already in progress: creating tablet
I20250625 13:52:20.231879 1171 ts_tablet_manager.cc:1397] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.124s user 0.002s sys 0.001s
W20250625 13:52:20.232219 711 leader_election.cc:343] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.232885 1045 raft_consensus.cc:3058] T 647c97a8ae484c4a8694e777bc62d602 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.235966 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c5a0db24bad64fdf840914eda05b9467" } modify_external_catalogs: true
I20250625 13:52:20.236131 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c5a0db24bad64fdf840914eda05b9467" } modify_external_catalogs: true
I20250625 13:52:20.237524 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:20.227264 1165 rpcz_store.cc:276] 0625 13:52:18.817973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.818036 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:20.226979 (+1408943us) ts_tablet_manager.cc:1145] Deleting tablet 7e4fbede3f9040709ae4bdc9bbb792ad
0625 13:52:20.226990 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.227087 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347513}
I20250625 13:52:20.238546 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1372) took 1390 ms. Trace:
I20250625 13:52:20.238620 1165 rpcz_store.cc:276] 0625 13:52:18.848344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.848653 (+ 309us) service_pool.cc:225] Handling call
0625 13:52:20.238434 (+1389781us) ts_tablet_manager.cc:1145] Deleting tablet f0a59da2129141a698692c4b31609d1d
0625 13:52:20.238445 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.238528 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355065}
I20250625 13:52:20.238845 1165 tablet_replica.cc:331] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:20.239289 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d9462da7574a46839763b1c388102e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:58 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.239552 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1258) took 4512 ms. Trace:
I20250625 13:52:20.239634 1168 rpcz_store.cc:276] 0625 13:52:15.718407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.718457 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:20.230688 (+4512231us) ts_tablet_manager.cc:1145] Deleting tablet 2e2711256c9042ebb651ae0ca3d02c89
0625 13:52:20.230699 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.230794 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4512062}
I20250625 13:52:20.240562 1165 raft_consensus.cc:2241] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.241262 1165 raft_consensus.cc:2270] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.241685 592 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet de3238f72d284c058f713260a802030f
W20250625 13:52:20.242972 711 leader_election.cc:343] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.243081 711 leader_election.cc:304] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:20.243271 2594 raft_consensus.cc:2747] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.244248 1165 ts_tablet_manager.cc:1905] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.254294 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c2bc65ae9c22425881fa52f654ed3541" } new_table_name: "default.Table_50a6dcd0a6a948ceb2f7ffb547f0be00" modify_external_catalogs: true
I20250625 13:52:20.256453 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1259) took 4536 ms. Trace:
I20250625 13:52:20.256541 1168 rpcz_store.cc:276] 0625 13:52:15.720417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.720495 (+ 78us) service_pool.cc:225] Handling call
0625 13:52:20.256294 (+4535799us) ts_tablet_manager.cc:1145] Deleting tablet bc0af0cf4cb54922ad6d1f5de12a46e0
0625 13:52:20.256306 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.256430 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535448}
I20250625 13:52:20.256811 1168 tablet_replica.cc:331] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.256981 1168 raft_consensus.cc:2241] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.257115 1168 raft_consensus.cc:2270] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.257957 1168 ts_tablet_manager.cc:1905] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.257946 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_7df890cf6e6d470485925ea0e2772f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.258459 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7df890cf6e6d470485925ea0e2772f5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 13:52:20.258909 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet 2e2711256c9042ebb651ae0ca3d02c89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e2711256c9042ebb651ae0ca3d02c89 already in progress: creating tablet
W20250625 13:52:20.259150 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet bc0af0cf4cb54922ad6d1f5de12a46e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc0af0cf4cb54922ad6d1f5de12a46e0 already in progress: creating tablet
I20250625 13:52:20.260587 1170 tablet_bootstrap.cc:492] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.260720 1170 ts_tablet_manager.cc:1397] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.002s
I20250625 13:52:20.260740 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_70e8ca6bcdff497daaf2b4dcc9a8183e" } modify_external_catalogs: true
I20250625 13:52:20.260902 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_70e8ca6bcdff497daaf2b4dcc9a8183e" } modify_external_catalogs: true
I20250625 13:52:20.261343 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7344157e71594339a43c2dd8d6e1fe58" } new_table_name: "default.Table_db71289018eb43f582db927c12500a1c" modify_external_catalogs: true
I20250625 13:52:20.261488 1170 raft_consensus.cc:357] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.261637 1170 raft_consensus.cc:383] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.261682 1170 raft_consensus.cc:738] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.261899 1170 consensus_queue.cc:260] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.262368 1170 ts_tablet_manager.cc:1428] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:20.262575 2254 ts_tablet_manager.cc:1918] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.262622 1170 tablet_bootstrap.cc:492] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.262663 2254 log.cc:1199] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a29708fb688a4a65b84ae2564a040b4c
I20250625 13:52:20.262753 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_e90bcf7a778340f286355f44c53e903c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:20.263012 2254 ts_tablet_manager.cc:1939] T a29708fb688a4a65b84ae2564a040b4c P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:20.263357 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e90bcf7a778340f286355f44c53e903c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.264256 1170 tablet_bootstrap.cc:654] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.264704 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d3e067c08fee43e69098a905cf6418aa" } modify_external_catalogs: true
I20250625 13:52:20.264874 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_d3e067c08fee43e69098a905cf6418aa" } modify_external_catalogs: true
I20250625 13:52:20.266253 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_6d7b4841a77e4c979f8edbff6abf9e55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.266805 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d7b4841a77e4c979f8edbff6abf9e55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.268462 2631 raft_consensus.cc:491] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.268590 2631 raft_consensus.cc:513] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:20.269598 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c605d5b64c164c1594b0a40cfa2c4cf6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.270190 849 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.270341 2631 leader_election.cc:290] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:20.278710 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc60ebdbced54d67b6cb712fc04ce85b consensus state has no leader
I20250625 13:52:20.281808 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7802866074204a24ba0302a41c6bbaf7" } modify_external_catalogs: true
I20250625 13:52:20.281980 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7802866074204a24ba0302a41c6bbaf7" } modify_external_catalogs: true
I20250625 13:52:20.282033 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_ae649101c4f34ca69ea84f0679906499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.282541 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae649101c4f34ca69ea84f0679906499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.282835 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_bb1cf312a582469abd7cf3cce0579710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.283273 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb1cf312a582469abd7cf3cce0579710 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.283639 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_faebb30460c84e39a94cee643bcc2309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.284009 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faebb30460c84e39a94cee643bcc2309 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.232595 1171 raft_consensus.cc:357] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:20.286978 1171 raft_consensus.cc:383] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.287081 1171 raft_consensus.cc:738] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.287374 1171 consensus_queue.cc:260] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:20.287918 1171 ts_tablet_manager.cc:1428] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20250625 13:52:20.290987 1171 tablet_bootstrap.cc:492] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.295008 1171 tablet_bootstrap.cc:654] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.300275 1170 tablet_bootstrap.cc:492] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.300536 1170 ts_tablet_manager.cc:1397] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250625 13:52:20.301702 1170 raft_consensus.cc:357] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.301944 1170 raft_consensus.cc:383] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.302043 1170 raft_consensus.cc:738] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.302275 1170 consensus_queue.cc:260] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.302909 1170 ts_tablet_manager.cc:1428] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:20.303254 1170 tablet_bootstrap.cc:492] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.305387 1170 tablet_bootstrap.cc:654] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.295624 2573 raft_consensus.cc:491] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.326042 2573 raft_consensus.cc:513] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.326915 2573 leader_election.cc:290] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:20.328732 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.329416 849 leader_election.cc:343] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.331969 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1317) took 2356 ms. Trace:
I20250625 13:52:20.332141 2254 rpcz_store.cc:276] 0625 13:52:17.975142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.975373 (+ 231us) service_pool.cc:225] Handling call
0625 13:52:19.976011 (+2000638us) ts_tablet_manager.cc:1145] Deleting tablet a29708fb688a4a65b84ae2564a040b4c
0625 13:52:19.976020 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.226206 (+250186us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.262343 (+ 36137us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.331941 (+ 69598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969220}
I20250625 13:52:20.332703 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a29708fb688a4a65b84ae2564a040b4c (table default.Table_f4ab201a21c4415fa8b6ac70e579492b [id=5846a492c29b4bb4b2cc8bbe93285f97]) successfully deleted
I20250625 13:52:20.333235 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1318) took 2356 ms. Trace:
I20250625 13:52:20.334033 2254 rpcz_store.cc:276] 0625 13:52:17.976952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.977016 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:20.333077 (+2356061us) ts_tablet_manager.cc:1145] Deleting tablet 8e4df287c95946e2beb94524073e5ee2
0625 13:52:20.333089 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.333216 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2326160}
W20250625 13:52:20.334864 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8e4df287c95946e2beb94524073e5ee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e4df287c95946e2beb94524073e5ee2 already in progress: creating tablet
I20250625 13:52:20.335290 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1319) took 2356 ms. Trace:
I20250625 13:52:20.340031 2254 rpcz_store.cc:276] 0625 13:52:17.979165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.980180 (+ 1015us) service_pool.cc:225] Handling call
0625 13:52:20.335187 (+2355007us) ts_tablet_manager.cc:1145] Deleting tablet 663588b7e9ee430faa180fec00897e22
0625 13:52:20.335197 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.335276 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2328179}
W20250625 13:52:20.343698 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 663588b7e9ee430faa180fec00897e22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 663588b7e9ee430faa180fec00897e22 already in progress: creating tablet
I20250625 13:52:20.344283 2254 tablet_replica.cc:331] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
W20250625 13:52:20.344936 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 304171310536435db21a6852b31ff208 consensus state has no leader
I20250625 13:52:20.345295 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_d6cb85bab6c44d7bb80f746165017e62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:20.346258 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9
I20250625 13:52:20.346349 2254 raft_consensus.cc:2241] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.346521 2254 pending_rounds.cc:70] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:20.346670 2254 pending_rounds.cc:77] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520669670170624 op_type: NO_OP noop_request { }
I20250625 13:52:20.346732 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 0b5ae05ecd414957a61d466a681d3016 from {username='slave'} at 127.0.0.1:54762
I20250625 13:52:20.346777 2254 raft_consensus.cc:2887] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:20.346871 2254 raft_consensus.cc:2270] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.346748 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d85834f366fa42c294c9290af21ea111: Sending DeleteTablet for 3 replicas of tablet e29bade90b184d2483cb1fe85033a03c
I20250625 13:52:20.347621 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_c979e1bc235d46debf85733939967c4d" } new_table_name: "default.Table_e46d21f14cf14f19b7944197845d51ea" modify_external_catalogs: true
I20250625 13:52:20.347654 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_58e294098e864cd5a9dd39dcbf23b881"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:52:20.348105 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58e294098e864cd5a9dd39dcbf23b881 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.348449 1170 tablet_bootstrap.cc:492] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.348609 1170 ts_tablet_manager.cc:1397] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20250625 13:52:20.349036 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_077db6683fc74d338d567587b5c87966"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:20.349195 2254 ts_tablet_manager.cc:1905] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.349363 1170 raft_consensus.cc:357] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:20.349488 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_077db6683fc74d338d567587b5c87966 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.349608 1170 raft_consensus.cc:383] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.349722 1170 raft_consensus.cc:738] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.349958 1170 consensus_queue.cc:260] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.350369 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e29bade90b184d2483cb1fe85033a03c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.350574 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.350831 878 tablet_service.cc:1515] Processing DeleteTablet for tablet db6e9db2375144b38ec43ffca9ca4c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:51:57 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:20.351044 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:20.351207 895 tablet_service.cc:1515] Processing DeleteTablet for tablet e29bade90b184d2483cb1fe85033a03c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:20.352288 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_35471795fa7d4688ae966d55ca175371" } modify_external_catalogs: true
I20250625 13:52:20.352423 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_35471795fa7d4688ae966d55ca175371" } modify_external_catalogs: true
W20250625 13:52:20.349689 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6cb85bab6c44d7bb80f746165017e62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.360044 1170 ts_tablet_manager.cc:1428] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 13:52:20.360360 1170 tablet_bootstrap.cc:492] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.362110 1170 tablet_bootstrap.cc:654] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.367105 591 catalog_manager.cc:5674] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291) does not have the latest schema for tablet 423ca098138c476f861a19b44554fd24 (table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]). Expected version 1 got 0
I20250625 13:52:20.369745 1168 ts_tablet_manager.cc:1918] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.369868 1168 log.cc:1199] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/9a2007e51d424053bc2836dedf46add2
I20250625 13:52:20.370209 1168 ts_tablet_manager.cc:1939] T 9a2007e51d424053bc2836dedf46add2 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:20.370944 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1264) took 4598 ms. Trace:
I20250625 13:52:20.371028 1168 rpcz_store.cc:276] 0625 13:52:15.772099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.772295 (+ 196us) service_pool.cc:225] Handling call
0625 13:52:20.256770 (+4484475us) ts_tablet_manager.cc:1145] Deleting tablet 9a2007e51d424053bc2836dedf46add2
0625 13:52:20.256778 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.322621 (+ 65843us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.369621 (+ 47000us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.370924 (+ 1303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4484293}
I20250625 13:52:20.377135 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet 9a2007e51d424053bc2836dedf46add2 (table default.Table_4a226ca5c5c84ef380d6c37dd08f6740 [id=fa06a5b0571b4a3a8b9b68f1f05fc3e6]) successfully deleted
I20250625 13:52:20.379191 1171 tablet_bootstrap.cc:492] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.379519 1171 ts_tablet_manager.cc:1397] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.001s
I20250625 13:52:20.380468 1171 raft_consensus.cc:357] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.380738 1171 raft_consensus.cc:383] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.380842 1171 raft_consensus.cc:738] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.381110 1171 consensus_queue.cc:260] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.382233 1171 ts_tablet_manager.cc:1428] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:20.386034 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_0a691ba3cecb4068b1955aa46a0ead68" } new_table_name: "default.Table_8362565cb5f34b5fb4e5cb94da328683" modify_external_catalogs: true
I20250625 13:52:20.388468 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d8985f259eb54ffdab29a9f6a2c5ae16 with delete_type TABLET_DATA_DELETED (replaced by tablet 879d91dcfb0b4868a7ba6833c211e5eb) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.390331 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1265) took 4618 ms. Trace:
I20250625 13:52:20.390471 1171 tablet_bootstrap.cc:492] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.390483 1168 rpcz_store.cc:276] 0625 13:52:15.772240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.772506 (+ 266us) service_pool.cc:225] Handling call
0625 13:52:20.371291 (+4598785us) ts_tablet_manager.cc:1145] Deleting tablet d5abd4d83514458c95ec62290b709bc3
0625 13:52:20.383078 (+ 11787us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.390309 (+ 7231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4598655}
W20250625 13:52:20.391299 581 catalog_manager.cc:4908] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): delete failed for tablet d5abd4d83514458c95ec62290b709bc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5abd4d83514458c95ec62290b709bc3 already in progress: creating tablet
I20250625 13:52:20.391706 1168 tablet_replica.cc:331] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.392233 1171 tablet_bootstrap.cc:654] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.391943 1168 raft_consensus.cc:2241] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.393769 1168 raft_consensus.cc:2270] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.394353 1168 ts_tablet_manager.cc:1905] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.396695 1170 tablet_bootstrap.cc:492] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.396884 1170 ts_tablet_manager.cc:1397] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20250625 13:52:20.397737 1170 raft_consensus.cc:357] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.397969 1170 raft_consensus.cc:383] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.398058 1170 raft_consensus.cc:738] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.398325 1170 consensus_queue.cc:260] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.399075 1170 ts_tablet_manager.cc:1428] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:20.399881 1170 tablet_bootstrap.cc:492] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.401880 1170 tablet_bootstrap.cc:654] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.424181 2594 raft_consensus.cc:491] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.424371 2594 raft_consensus.cc:513] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.424998 2594 leader_election.cc:290] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.425859 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.426060 914 raft_consensus.cc:2466] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:20.426455 711 leader_election.cc:304] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:20.431607 1168 ts_tablet_manager.cc:1918] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.405076 2254 ts_tablet_manager.cc:1918] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:20.436151 2254 log.cc:1199] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/63d01af6b320489fb99c467c11ed1dc0
I20250625 13:52:20.436610 2254 ts_tablet_manager.cc:1939] T 63d01af6b320489fb99c467c11ed1dc0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:20.437078 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1320) took 2240 ms. Trace:
I20250625 13:52:20.437237 2254 rpcz_store.cc:276] 0625 13:52:18.196929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.197009 (+ 80us) service_pool.cc:225] Handling call
0625 13:52:20.344221 (+2147212us) ts_tablet_manager.cc:1145] Deleting tablet 63d01af6b320489fb99c467c11ed1dc0
0625 13:52:20.344234 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.373865 (+ 29631us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.404972 (+ 31107us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.437058 (+ 32086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2146964}
I20250625 13:52:20.435209 2636 consensus_queue.cc:1035] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:20.431440 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:20.441855 1045 raft_consensus.cc:2466] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:20.442659 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet 63d01af6b320489fb99c467c11ed1dc0 (table default.Table_a0109f4a8b5a469d812ec2c8fe30b5ae [id=c4babc967a48425587250196c8d9b4ab]) successfully deleted
I20250625 13:52:20.443697 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1321) took 2173 ms. Trace:
I20250625 13:52:20.444834 2254 rpcz_store.cc:276] 0625 13:52:18.269800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.269853 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:20.443530 (+2173677us) ts_tablet_manager.cc:1145] Deleting tablet d92fe17f576e491eb189a10ca72a7e09
0625 13:52:20.443545 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.443675 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173472}
W20250625 13:52:20.447297 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet d92fe17f576e491eb189a10ca72a7e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d92fe17f576e491eb189a10ca72a7e09 already in progress: creating tablet
I20250625 13:52:20.447839 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1322) took 2176 ms. Trace:
I20250625 13:52:20.448244 2254 rpcz_store.cc:276] 0625 13:52:18.270962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.271021 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:20.447703 (+2176682us) ts_tablet_manager.cc:1145] Deleting tablet 6c89a899837f40a8ac9eed9f1d68e420
0625 13:52:20.447723 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.447824 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2176512}
W20250625 13:52:20.448947 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 6c89a899837f40a8ac9eed9f1d68e420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c89a899837f40a8ac9eed9f1d68e420 already in progress: creating tablet
I20250625 13:52:20.449378 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1323) took 2162 ms. Trace:
I20250625 13:52:20.449733 2254 rpcz_store.cc:276] 0625 13:52:18.287334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.287398 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:20.449274 (+2161876us) ts_tablet_manager.cc:1145] Deleting tablet 8a9b6e6e1f2d41288a958115dc18d4fb
0625 13:52:20.449284 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.449361 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2161625}
W20250625 13:52:20.450311 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8a9b6e6e1f2d41288a958115dc18d4fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a9b6e6e1f2d41288a958115dc18d4fb already in progress: creating tablet
I20250625 13:52:20.450872 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1324) took 2162 ms. Trace:
I20250625 13:52:20.452112 2254 rpcz_store.cc:276] 0625 13:52:18.288398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.288450 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:20.450752 (+2162302us) ts_tablet_manager.cc:1145] Deleting tablet 8dd0d216fb394f31afa4478ad2097fe5
0625 13:52:20.450762 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.450858 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162076}
W20250625 13:52:20.452941 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 8dd0d216fb394f31afa4478ad2097fe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dd0d216fb394f31afa4478ad2097fe5 already in progress: creating tablet
I20250625 13:52:20.453370 2254 tablet_replica.cc:331] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:20.453778 2254 raft_consensus.cc:2241] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:20.454386 2254 raft_consensus.cc:2270] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.460103 2594 raft_consensus.cc:2802] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:20.460222 2594 raft_consensus.cc:491] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:20.460290 2594 raft_consensus.cc:3058] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:20.461434 985 consensus_peers.cc:487] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69672579a83c41d09ee7c78a6fa7d746. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:20.460925 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_3ba7e48a0b6d4800b85ad1345163f099" } new_table_name: "default.Table_d57d9bc16e08492bba84ebd6c4186a0b" modify_external_catalogs: true
I20250625 13:52:20.464439 2254 ts_tablet_manager.cc:1905] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.461160 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e3ef7f5b474466bbc19fd31c6b6d99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:01 UTC) from {username='slave'} at 127.0.0.1:52994
I20250625 13:52:20.468489 1168 log.cc:1199] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/da2f8d0d9a714a28ba6fb9642d461de2
I20250625 13:52:20.469097 1168 ts_tablet_manager.cc:1939] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:20.477278 1171 tablet_bootstrap.cc:492] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.477494 1171 ts_tablet_manager.cc:1397] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.006s
I20250625 13:52:20.478312 1171 raft_consensus.cc:357] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.478588 1171 raft_consensus.cc:383] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.478720 1171 raft_consensus.cc:738] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.478996 1171 consensus_queue.cc:260] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.479627 1171 ts_tablet_manager.cc:1428] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:20.479919 1171 tablet_bootstrap.cc:492] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.481315 2594 raft_consensus.cc:513] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.481926 2594 leader_election.cc:290] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.487921 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:20.488145 916 raft_consensus.cc:3058] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.489975 1171 tablet_bootstrap.cc:654] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.491132 916 raft_consensus.cc:2466] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:20.491812 711 leader_election.cc:304] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:20.492611 2594 raft_consensus.cc:2802] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:20.498298 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:20.498581 1045 raft_consensus.cc:3058] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.499228 1165 ts_tablet_manager.cc:1918] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:20.499392 1165 log.cc:1199] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/653267b468ef4958b98af0ed6a7bf4d3
I20250625 13:52:20.499866 1165 ts_tablet_manager.cc:1939] T 653267b468ef4958b98af0ed6a7bf4d3 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:20.499271 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54762:
name: "default.Table_8b85822eacbd47aa89529bc6fcf3ee29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:52:20.507103 764 raft_consensus.cc:2466] T ed3c177bb6dd488aa54bee2a447016ff P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:20.507670 758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43474 (request call id 1415) took 1046 ms. Trace:
I20250625 13:52:20.508502 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1373) took 1642 ms. Trace:
I20250625 13:52:20.508653 1165 rpcz_store.cc:276] 0625 13:52:18.866443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.866516 (+ 73us) service_pool.cc:225] Handling call
0625 13:52:20.238810 (+1372294us) ts_tablet_manager.cc:1145] Deleting tablet 653267b468ef4958b98af0ed6a7bf4d3
0625 13:52:20.238817 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.495812 (+256995us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.499131 (+ 3319us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.508482 (+ 9351us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2270,"tablet-delete.queue_time_us":1348098}
I20250625 13:52:20.509186 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1374) took 1642 ms. Trace:
I20250625 13:52:20.509305 1165 rpcz_store.cc:276] 0625 13:52:18.866959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.867013 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:20.509059 (+1642046us) ts_tablet_manager.cc:1145] Deleting tablet a89db9678dd444d5a4df85baccd62044
0625 13:52:20.509071 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.509172 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615317}
I20250625 13:52:20.511492 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1393) took 1190 ms. Trace:
I20250625 13:52:20.511621 1165 rpcz_store.cc:276] 0625 13:52:19.321299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.321496 (+ 197us) service_pool.cc:225] Handling call
0625 13:52:20.511381 (+1189885us) ts_tablet_manager.cc:1145] Deleting tablet 7e8396ea40d34c66ab740c66647a6afc
0625 13:52:20.511393 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.511476 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1189595}
I20250625 13:52:20.512004 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1394) took 1162 ms. Trace:
I20250625 13:52:20.512136 1165 rpcz_store.cc:276] 0625 13:52:19.349820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.349873 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:20.511910 (+1162037us) ts_tablet_manager.cc:1145] Deleting tablet cbd82bb415174dbdbcfafa9d6c3f3aa7
0625 13:52:20.511918 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.511989 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1139923}
I20250625 13:52:20.512390 1171 tablet_bootstrap.cc:492] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.512492 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1395) took 1160 ms. Trace:
I20250625 13:52:20.512696 1171 ts_tablet_manager.cc:1397] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.001s
I20250625 13:52:20.512622 1165 rpcz_store.cc:276] 0625 13:52:19.351552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.351620 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:20.512403 (+1160783us) ts_tablet_manager.cc:1145] Deleting tablet 7e8396ea40d34c66ab740c66647a6afc
0625 13:52:20.512411 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.512479 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140340}
I20250625 13:52:20.507764 758 rpcz_store.cc:276] 0625 13:52:19.461662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.461706 (+ 44us) service_pool.cc:225] Handling call
0625 13:52:19.462022 (+ 316us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0625 13:52:19.462041 (+ 19us) ts_tablet_manager.cc:676] Creating new metadata...
0625 13:52:20.136654 (+674613us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.507483 (+370829us) tablet_replica.cc:206] Creating consensus instance
0625 13:52:20.507654 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:52:20.512923 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1396) took 1154 ms. Trace:
I20250625 13:52:20.513000 1165 rpcz_store.cc:276] 0625 13:52:19.358921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.358993 (+ 72us) service_pool.cc:225] Handling call
0625 13:52:20.512832 (+1153839us) ts_tablet_manager.cc:1145] Deleting tablet d8257f79a42247b48f2736a545362f44
0625 13:52:20.512842 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.512910 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140638}
I20250625 13:52:20.513274 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1403) took 1072 ms. Trace:
I20250625 13:52:20.513347 1165 rpcz_store.cc:276] 0625 13:52:19.440724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.441047 (+ 323us) service_pool.cc:225] Handling call
0625 13:52:20.513190 (+1072143us) ts_tablet_manager.cc:1145] Deleting tablet e1549b9e797a44f89b316a44c5e7ee2e
0625 13:52:20.513198 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.513261 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1071986}
I20250625 13:52:20.513631 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1404) took 1072 ms. Trace:
I20250625 13:52:20.514218 1171 raft_consensus.cc:357] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.514468 1171 raft_consensus.cc:383] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.514592 1171 raft_consensus.cc:738] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.514891 1171 consensus_queue.cc:260] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.516106 1171 ts_tablet_manager.cc:1428] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:20.516628 1166 tablet_bootstrap.cc:654] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.517292 1171 tablet_bootstrap.cc:492] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.519140 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1270) took 4690 ms. Trace:
I20250625 13:52:20.519212 1168 rpcz_store.cc:276] 0625 13:52:15.828707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.828765 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:20.391659 (+4562894us) ts_tablet_manager.cc:1145] Deleting tablet da2f8d0d9a714a28ba6fb9642d461de2
0625 13:52:20.391668 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.408212 (+ 16544us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.431469 (+ 23257us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.519119 (+ 87650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4522630}
I20250625 13:52:20.519687 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet da2f8d0d9a714a28ba6fb9642d461de2 (table default.Table_dc86aa488ba944c689ae858bc3800718 [id=028f5254a2c045929bdb41c0b273875c]) successfully deleted
I20250625 13:52:20.520056 1168 tablet_replica.cc:331] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.520301 1168 raft_consensus.cc:2241] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.520605 1168 raft_consensus.cc:2270] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.521193 1168 ts_tablet_manager.cc:1905] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.524340 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd164b54c1d14ea98422360705cf42bf" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:52:20.527443 851 leader_election.cc:343] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1ca3f3760e6e41a29d004e6afe63ec98
I20250625 13:52:20.527567 851 leader_election.cc:304] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:20.527619 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet aafb7eef813f48fc8295fbb30a29cd0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aafb7eef813f48fc8295fbb30a29cd0d already in progress: creating tablet
W20250625 13:52:20.527904 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7e4fbede3f9040709ae4bdc9bbb792ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e4fbede3f9040709ae4bdc9bbb792ad already in progress: creating tablet
W20250625 13:52:20.527936 851 leader_election.cc:343] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d9462da7574a46839763b1c388102e60
I20250625 13:52:20.528018 851 leader_election.cc:304] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:20.528113 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet f0a59da2129141a698692c4b31609d1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0a59da2129141a698692c4b31609d1d already in progress: creating tablet
I20250625 13:52:20.528302 2573 raft_consensus.cc:2747] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.528443 2573 raft_consensus.cc:2747] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.530661 1171 tablet_bootstrap.cc:654] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:20.530966 851 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: da2f8d0d9a714a28ba6fb9642d461de2
I20250625 13:52:20.531068 851 leader_election.cc:304] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:20.531210 851 raft_consensus.cc:2684] T da2f8d0d9a714a28ba6fb9642d461de2 P affa1c3035b940e88ca143be7d35840d: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 13:52:20.531850 851 leader_election.cc:343] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7506cce0ec4c4ec68f25591cb98b1252
I20250625 13:52:20.531949 851 leader_election.cc:304] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.532140 2631 raft_consensus.cc:2747] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:20.532306 851 consensus_peers.cc:487] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:20.532611 851 leader_election.cc:304] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:20.532853 2631 raft_consensus.cc:2802] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:20.532965 2631 raft_consensus.cc:695] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
W20250625 13:52:20.533125 851 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:52:20.533200 851 leader_election.cc:304] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.533233 2631 consensus_queue.cc:237] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.533905 2204 raft_consensus.cc:2747] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.534945 583 catalog_manager.cc:4928] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): tablet 653267b468ef4958b98af0ed6a7bf4d3 (table default.Table_107a64d05b814c5a9cc7b956f38af315 [id=6bfad46432b7473f88d77bfcfa2a4e9a]) successfully deleted
W20250625 13:52:20.535215 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet a89db9678dd444d5a4df85baccd62044 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a89db9678dd444d5a4df85baccd62044 already in progress: creating tablet
W20250625 13:52:20.535499 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7e8396ea40d34c66ab740c66647a6afc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e8396ea40d34c66ab740c66647a6afc already in progress: creating tablet
W20250625 13:52:20.535691 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbd82bb415174dbdbcfafa9d6c3f3aa7 already in progress: creating tablet
W20250625 13:52:20.535863 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet 7e8396ea40d34c66ab740c66647a6afc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e8396ea40d34c66ab740c66647a6afc already in progress: creating tablet
I20250625 13:52:20.536101 1170 tablet_bootstrap.cc:492] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:20.536151 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet d8257f79a42247b48f2736a545362f44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8257f79a42247b48f2736a545362f44 already in progress: creating tablet
I20250625 13:52:20.536224 1170 ts_tablet_manager.cc:1397] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.136s user 0.006s sys 0.008s
W20250625 13:52:20.536376 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e1549b9e797a44f89b316a44c5e7ee2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1549b9e797a44f89b316a44c5e7ee2e already in progress: creating tablet
I20250625 13:52:20.536906 1170 raft_consensus.cc:357] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:20.537122 1170 raft_consensus.cc:383] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.537206 1170 raft_consensus.cc:738] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.537489 1170 consensus_queue.cc:260] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:20.537670 2674 tablet.cc:1722] T d7c6a325d40c4f7ba2f342e39b9437e3 P 5c1aa07ab6bc4b0089e46951240a37c9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 13:52:20.544363 1170 ts_tablet_manager.cc:1428] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250625 13:52:20.547284 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b85822eacbd47aa89529bc6fcf3ee29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:52:20.549440 589 catalog_manager.cc:5674] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801) does not have the latest schema for tablet d7c6a325d40c4f7ba2f342e39b9437e3 (table default.Table_727a4473adab487fa65f094237ec794b [id=3977ee3968d246b7a8aed1fafd1822b7]). Expected version 3 got 0
I20250625 13:52:20.549918 1170 tablet_bootstrap.cc:492] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.551129 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:43474
I20250625 13:52:20.551558 758 tablet_service.cc:1515] Processing DeleteTablet for tablet e29bade90b184d2483cb1fe85033a03c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:20 UTC) from {username='slave'} at 127.0.0.1:43474
W20250625 13:52:20.566011 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5a50381bcaa4c51b6f6f5aa47237115 consensus state has no leader
I20250625 13:52:20.567912 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6180acf99695440fbee78730755721b3" } modify_external_catalogs: true
I20250625 13:52:20.568087 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_6180acf99695440fbee78730755721b3" } modify_external_catalogs: true
I20250625 13:52:20.568444 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_2009356d754d48c9a01a3058fe1ac4c5" } new_table_name: "default.Table_4415b2c037384931880ac63f3c7a3e7b" modify_external_catalogs: true
W20250625 13:52:20.570780 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c571f5508ab481b93e71babb1f65e78 consensus state has no leader
I20250625 13:52:20.571139 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_7222541aef894753bea095d602e31b74" } new_table_name: "default.Table_1d55f42f2cf24f078b5b2061b74d4e18" modify_external_catalogs: true
W20250625 13:52:20.573136 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fb7d9505d814d048088600fe14a51c1 has no consensus state
I20250625 13:52:20.573462 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1e884456a1d24e429f5e791a07f1014f" } modify_external_catalogs: true
I20250625 13:52:20.573604 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54762:
table { table_name: "default.Table_1e884456a1d24e429f5e791a07f1014f" } modify_external_catalogs: true
I20250625 13:52:20.573324 1168 ts_tablet_manager.cc:1918] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.574190 1168 log.cc:1199] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/dcf7fab53cae49908d2952cdaa07943a
I20250625 13:52:20.578100 1170 tablet_bootstrap.cc:654] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.579041 2254 ts_tablet_manager.cc:1918] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:20.579208 2254 log.cc:1199] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/a30a35e3639a43c5b078169060824723
I20250625 13:52:20.579721 2254 ts_tablet_manager.cc:1939] T a30a35e3639a43c5b078169060824723 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:20.580082 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1325) took 2289 ms. Trace:
I20250625 13:52:20.580216 2254 rpcz_store.cc:276] 0625 13:52:18.290140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.290186 (+ 46us) service_pool.cc:225] Handling call
0625 13:52:20.453316 (+2163130us) ts_tablet_manager.cc:1145] Deleting tablet a30a35e3639a43c5b078169060824723
0625 13:52:20.453332 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.518762 (+ 65430us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.578937 (+ 60175us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.580066 (+ 1129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162960}
I20250625 13:52:20.580766 581 catalog_manager.cc:4928] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): tablet a30a35e3639a43c5b078169060824723 (table default.Table_d7faba35e83944d586e1e53d4b524950 [id=33400d976688476eb4450525402f12be]) successfully deleted
I20250625 13:52:20.581225 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1328) took 1946 ms. Trace:
I20250625 13:52:20.581672 1168 ts_tablet_manager.cc:1939] T dcf7fab53cae49908d2952cdaa07943a P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:20.585040 2594 raft_consensus.cc:695] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:20.585402 2594 consensus_queue.cc:237] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.582469 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet c605d5b64c164c1594b0a40cfa2c4cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 13:52:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20250625 13:52:20.585547 2638 raft_consensus.cc:491] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.589898 2638 raft_consensus.cc:513] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.589953 2254 rpcz_store.cc:276] 0625 13:52:18.634702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.634752 (+ 50us) service_pool.cc:225] Handling call
0625 13:52:20.581085 (+1946333us) ts_tablet_manager.cc:1145] Deleting tablet 2191b836bb3044c89fee13c869d9e07d
0625 13:52:20.581094 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.581209 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1946138}
I20250625 13:52:20.584091 1171 tablet_bootstrap.cc:492] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.590481 2638 leader_election.cc:290] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.590618 1171 ts_tablet_manager.cc:1397] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.002s
I20250625 13:52:20.591027 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:20.591228 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423ca098138c476f861a19b44554fd24" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.591743 983 leader_election.cc:304] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:20.591791 1171 raft_consensus.cc:357] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.592013 1171 raft_consensus.cc:383] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.513734 1165 rpcz_store.cc:276] 0625 13:52:19.440981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.441234 (+ 253us) service_pool.cc:225] Handling call
0625 13:52:20.513547 (+1072313us) ts_tablet_manager.cc:1145] Deleting tablet e58128254acf44129f7b4e452ac03fe5
0625 13:52:20.513554 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.513617 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072189}
I20250625 13:52:20.592098 2638 raft_consensus.cc:2747] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.592111 1171 raft_consensus.cc:738] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.592432 1171 consensus_queue.cc:260] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:20.592588 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet e58128254acf44129f7b4e452ac03fe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e58128254acf44129f7b4e452ac03fe5 already in progress: creating tablet
I20250625 13:52:20.593508 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1405) took 1152 ms. Trace:
I20250625 13:52:20.593668 1165 rpcz_store.cc:276] 0625 13:52:19.441457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.442072 (+ 615us) service_pool.cc:225] Handling call
0625 13:52:20.593347 (+1151275us) ts_tablet_manager.cc:1145] Deleting tablet fc086d416d0f475987bd67f122487353
0625 13:52:20.593360 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.593471 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151134}
W20250625 13:52:20.594429 583 catalog_manager.cc:4908] TS 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): delete failed for tablet fc086d416d0f475987bd67f122487353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc086d416d0f475987bd67f122487353 already in progress: creating tablet
I20250625 13:52:20.595291 1171 ts_tablet_manager.cc:1428] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 13:52:20.596314 1165 tablet_replica.cc:331] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:20.596633 1165 raft_consensus.cc:2241] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.596216 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1329) took 1961 ms. Trace:
I20250625 13:52:20.597295 2254 rpcz_store.cc:276] 0625 13:52:18.634833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.634987 (+ 154us) service_pool.cc:225] Handling call
0625 13:52:20.594749 (+1959762us) ts_tablet_manager.cc:1145] Deleting tablet 391b412dbb9e420c90aa79473bbbe8d7
0625 13:52:20.594760 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.595872 (+ 1112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958438}
I20250625 13:52:20.598384 1171 tablet_bootstrap.cc:492] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.600409 1170 tablet_bootstrap.cc:492] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.600641 1170 ts_tablet_manager.cc:1397] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.004s
I20250625 13:52:20.601044 1165 raft_consensus.cc:2270] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.601349 1170 raft_consensus.cc:357] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.601599 1170 raft_consensus.cc:383] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.601701 1170 raft_consensus.cc:738] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.601768 1165 ts_tablet_manager.cc:1905] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.601981 1170 consensus_queue.cc:260] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.602420 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1330) took 1967 ms. Trace:
I20250625 13:52:20.602622 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1268) took 4774 ms. Trace:
I20250625 13:52:20.603063 2254 rpcz_store.cc:276] 0625 13:52:18.634940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.635141 (+ 201us) service_pool.cc:225] Handling call
0625 13:52:20.599974 (+1964833us) ts_tablet_manager.cc:1145] Deleting tablet 45079deff84a41b488ccdde2f5059431
0625 13:52:20.599981 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.602114 (+ 2133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964717}
I20250625 13:52:20.603482 1168 rpcz_store.cc:276] 0625 13:52:15.827693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.827752 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:20.520008 (+4692256us) ts_tablet_manager.cc:1145] Deleting tablet dcf7fab53cae49908d2952cdaa07943a
0625 13:52:20.520017 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.524572 (+ 4555us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.571975 (+ 47403us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.602318 (+ 30343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4650548}
I20250625 13:52:20.607234 1170 ts_tablet_manager.cc:1428] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:20.608095 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1331) took 1971 ms. Trace:
I20250625 13:52:20.608232 1168 tablet_replica.cc:331] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.608724 2254 rpcz_store.cc:276] 0625 13:52:18.636736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.636795 (+ 59us) service_pool.cc:225] Handling call
0625 13:52:20.605393 (+1968598us) ts_tablet_manager.cc:1145] Deleting tablet 3fc03dbe0a19470fbd0fe550159cd232
0625 13:52:20.605403 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.607799 (+ 2396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968244}
I20250625 13:52:20.610404 1168 raft_consensus.cc:2241] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.611788 1170 tablet_bootstrap.cc:492] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.612102 2254 tablet_replica.cc:331] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:20.612725 1168 raft_consensus.cc:2270] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.615394 2254 raft_consensus.cc:2241] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:20.617553 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu pid 563
I20250625 13:52:20.618214 1170 tablet_bootstrap.cc:654] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:20.618233 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 23a2b56c117f49d3b7bab4bb3d776173 (table default.Table_e17cf60c94784737b39faf63421fff96 [id=c41c875b733941ba83261e1e9853583c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 23a2b56c117f49d3b7bab4bb3d776173 consensus state has no leader
W20250625 13:52:20.590351 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 2191b836bb3044c89fee13c869d9e07d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2191b836bb3044c89fee13c869d9e07d already in progress: creating tablet
I20250625 13:52:20.619033 581 catalog_manager.cc:4928] TS affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): tablet dcf7fab53cae49908d2952cdaa07943a (table default.Table_fe872dae10544c0ca6a6735266cd03d2 [id=aa4ef3fe7bbb408396878d79f7caffb9]) successfully deleted
I20250625 13:52:20.619189 627 generic_service.cc:196] Checking for leaks (request via RPC)
W20250625 13:52:20.619247 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 391b412dbb9e420c90aa79473bbbe8d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 391b412dbb9e420c90aa79473bbbe8d7 already in progress: creating tablet
W20250625 13:52:20.619436 581 catalog_manager.cc:4908] TS 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): delete failed for tablet 45079deff84a41b488ccdde2f5059431 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45079deff84a41b488ccdde2f5059431 already in progress: creating tablet
I20250625 13:52:20.620088 1168 ts_tablet_manager.cc:1905] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.620548 2254 raft_consensus.cc:2270] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.625988 2254 ts_tablet_manager.cc:1905] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.630231 1170 tablet_bootstrap.cc:492] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.630370 1171 tablet_bootstrap.cc:654] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.630424 1170 ts_tablet_manager.cc:1397] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20250625 13:52:20.631141 1170 raft_consensus.cc:357] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.631368 1170 raft_consensus.cc:383] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.631455 1170 raft_consensus.cc:738] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.631735 1170 consensus_queue.cc:260] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.636720 1170 ts_tablet_manager.cc:1428] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 13:52:20.638525 1165 ts_tablet_manager.cc:1918] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.638638 1165 log.cc:1199] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/69c44d72516e4883aff636ca65929c11
I20250625 13:52:20.638907 1165 ts_tablet_manager.cc:1939] T 69c44d72516e4883aff636ca65929c11 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:20.640805 1170 tablet_bootstrap.cc:492] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.641500 1166 tablet_bootstrap.cc:492] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:20.641703 1166 ts_tablet_manager.cc:1397] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.492s user 0.003s sys 0.012s
I20250625 13:52:20.642542 1170 tablet_bootstrap.cc:654] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.642477 1166 raft_consensus.cc:357] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.642714 1166 raft_consensus.cc:383] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.642803 1166 raft_consensus.cc:738] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.643061 1166 consensus_queue.cc:260] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.643698 1166 ts_tablet_manager.cc:1428] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:20.640615 851 leader_election.cc:304] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.644342 2580 raft_consensus.cc:2747] T fd164b54c1d14ea98422360705cf42bf P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.657161 1171 tablet_bootstrap.cc:492] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.657406 1171 ts_tablet_manager.cc:1397] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.002s
I20250625 13:52:20.658228 1171 raft_consensus.cc:357] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.658478 1171 raft_consensus.cc:383] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.658583 1171 raft_consensus.cc:738] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.658835 1171 consensus_queue.cc:260] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.659420 1171 ts_tablet_manager.cc:1428] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:20.659785 1171 tablet_bootstrap.cc:492] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.661628 1171 tablet_bootstrap.cc:654] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.665885 2254 ts_tablet_manager.cc:1918] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:20.666087 2254 log.cc:1199] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/69672579a83c41d09ee7c78a6fa7d746
I20250625 13:52:20.666556 2254 ts_tablet_manager.cc:1939] T 69672579a83c41d09ee7c78a6fa7d746 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:20.668301 1166 tablet_bootstrap.cc:492] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:20.674805 1171 tablet_bootstrap.cc:492] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.674997 1171 ts_tablet_manager.cc:1397] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20250625 13:52:20.675755 1171 raft_consensus.cc:357] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.676003 1171 raft_consensus.cc:383] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.676095 1171 raft_consensus.cc:738] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.676371 1171 consensus_queue.cc:260] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.676887 1171 ts_tablet_manager.cc:1428] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:20.677281 1166 tablet_bootstrap.cc:654] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.677670 1171 tablet_bootstrap.cc:492] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.679132 1171 tablet_bootstrap.cc:654] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.682484 1168 ts_tablet_manager.cc:1918] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:20.685734 1168 log.cc:1199] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/69672579a83c41d09ee7c78a6fa7d746
I20250625 13:52:20.686163 1168 ts_tablet_manager.cc:1939] T 69672579a83c41d09ee7c78a6fa7d746 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:20.667991 985 consensus_peers.cc:487] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e2bc4df39674dc3be0cd32c68f1ebca. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:20.670104 1170 tablet_bootstrap.cc:492] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.688378 1170 ts_tablet_manager.cc:1397] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20250625 13:52:20.689100 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1347) took 1940 ms. Trace:
I20250625 13:52:20.689644 1170 raft_consensus.cc:357] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.690403 1170 raft_consensus.cc:383] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.690498 1170 raft_consensus.cc:738] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.690768 1170 consensus_queue.cc:260] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.691373 1170 ts_tablet_manager.cc:1428] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:20.691495 2254 rpcz_store.cc:276] 0625 13:52:18.748428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.750653 (+ 2225us) service_pool.cc:225] Handling call
0625 13:52:20.611388 (+1860735us) ts_tablet_manager.cc:1145] Deleting tablet 69672579a83c41d09ee7c78a6fa7d746
0625 13:52:20.611778 (+ 390us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.632816 (+ 21038us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.665756 (+ 32940us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.689074 (+ 23318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1860530}
I20250625 13:52:20.691877 1165 tablet_replica.cc:331] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:20.680651 1045 raft_consensus.cc:2466] T fd164b54c1d14ea98422360705cf42bf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:20.692201 1165 raft_consensus.cc:2241] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.692440 1165 raft_consensus.cc:2270] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.692816 1170 tablet_bootstrap.cc:492] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.693233 1165 ts_tablet_manager.cc:1905] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.694865 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1348) took 1894 ms. Trace:
I20250625 13:52:20.695060 2254 rpcz_store.cc:276] 0625 13:52:18.800823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.809424 (+ 8601us) service_pool.cc:225] Handling call
0625 13:52:20.691978 (+1882554us) ts_tablet_manager.cc:1145] Deleting tablet aafb7eef813f48fc8295fbb30a29cd0d
0625 13:52:20.692919 (+ 941us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.694845 (+ 1926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882325}
I20250625 13:52:20.700498 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1269) took 4872 ms. Trace:
I20250625 13:52:20.701138 1168 rpcz_store.cc:276] 0625 13:52:15.828297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.828352 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:20.607904 (+4779552us) ts_tablet_manager.cc:1145] Deleting tablet 69672579a83c41d09ee7c78a6fa7d746
0625 13:52:20.607917 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.646725 (+ 38808us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.682388 (+ 35663us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.700472 (+ 18084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4737405}
I20250625 13:52:20.701706 1168 tablet_replica.cc:331] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.701965 1168 raft_consensus.cc:2241] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.702167 1168 raft_consensus.cc:2270] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.702293 1166 tablet_bootstrap.cc:492] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:20.702469 1166 ts_tablet_manager.cc:1397] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250625 13:52:20.702922 1168 ts_tablet_manager.cc:1905] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.703387 1166 raft_consensus.cc:357] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.703605 1166 raft_consensus.cc:383] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.703703 1166 raft_consensus.cc:738] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.703971 1166 consensus_queue.cc:260] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.713205 2254 tablet_replica.cc:331] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:20.713488 2254 raft_consensus.cc:2241] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:20.713757 2254 pending_rounds.cc:70] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:20.713882 2254 pending_rounds.cc:77] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520675337326592 op_type: NO_OP noop_request { }
I20250625 13:52:20.715179 2594 raft_consensus.cc:491] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.715343 2594 raft_consensus.cc:513] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.715660 2254 raft_consensus.cc:2887] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:20.715757 2254 raft_consensus.cc:2270] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.715934 2594 leader_election.cc:290] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.716560 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2e6fbf3d81547f798b246a8b8f61387" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.716750 915 raft_consensus.cc:2466] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:20.717365 711 leader_election.cc:304] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:20.717610 2594 raft_consensus.cc:2802] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:20.717684 2594 raft_consensus.cc:491] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:20.717767 2594 raft_consensus.cc:3058] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.719470 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2e6fbf3d81547f798b246a8b8f61387" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.719947 711 leader_election.cc:343] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.721463 2594 raft_consensus.cc:513] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.722388 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2e6fbf3d81547f798b246a8b8f61387" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:20.722855 711 leader_election.cc:343] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.723134 2693 raft_consensus.cc:491] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.723243 2693 raft_consensus.cc:513] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.723516 1170 tablet_bootstrap.cc:654] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.723917 2254 ts_tablet_manager.cc:1905] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.723937 2693 leader_election.cc:290] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.724638 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "189a9347cd5d4b5f955e5c0706aa3947" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.724642 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "189a9347cd5d4b5f955e5c0706aa3947" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.725085 711 leader_election.cc:343] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:20.725250 711 leader_election.cc:343] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.725322 711 leader_election.cc:304] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:20.725840 2693 raft_consensus.cc:2747] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.726290 2594 leader_election.cc:290] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.726517 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2e6fbf3d81547f798b246a8b8f61387" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:20.726665 915 raft_consensus.cc:3058] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.726781 1166 ts_tablet_manager.cc:1428] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.024s user 0.000s sys 0.001s
I20250625 13:52:20.729557 915 raft_consensus.cc:2466] T e2e6fbf3d81547f798b246a8b8f61387 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:20.731369 711 leader_election.cc:304] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:20.732169 2594 raft_consensus.cc:2802] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:20.732811 2594 raft_consensus.cc:695] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:20.733167 2594 consensus_queue.cc:237] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.734488 1166 tablet_bootstrap.cc:492] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:20.736160 1166 tablet_bootstrap.cc:654] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.738811 2254 ts_tablet_manager.cc:1918] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250625 13:52:20.738240 743 consensus_meta.cc:287] T 8146355663ad48469756034e35a081ef P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent flushing consensus metadata: real 0.587s user 0.024s sys 0.000s
I20250625 13:52:20.742233 743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43474 (request call id 1432) took 1011 ms. Trace:
I20250625 13:52:20.747061 1168 ts_tablet_manager.cc:1918] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.747802 743 rpcz_store.cc:276] 0625 13:52:19.731078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.963483 (+232405us) service_pool.cc:225] Handling call
0625 13:52:19.963866 (+ 383us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0625 13:52:20.031743 (+ 67877us) ts_tablet_manager.cc:676] Creating new metadata...
0625 13:52:20.151177 (+119434us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.741786 (+590609us) tablet_replica.cc:206] Creating consensus instance
0625 13:52:20.742212 (+ 426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 13:52:20.756763 1166 tablet_bootstrap.cc:492] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:20.760749 1166 ts_tablet_manager.cc:1397] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250625 13:52:20.761595 1166 raft_consensus.cc:357] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.761848 1166 raft_consensus.cc:383] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.761859 1168 log.cc:1199] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/2de9656958164dc4ae2fb0078b38f6c3
I20250625 13:52:20.761082 1171 tablet_bootstrap.cc:492] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.762334 1166 raft_consensus.cc:738] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.762403 1171 ts_tablet_manager.cc:1397] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.005s
I20250625 13:52:20.762498 1168 ts_tablet_manager.cc:1939] T 2de9656958164dc4ae2fb0078b38f6c3 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:20.762704 1166 consensus_queue.cc:260] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.763132 1171 raft_consensus.cc:357] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.769340 1171 raft_consensus.cc:383] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.769840 1166 ts_tablet_manager.cc:1428] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250625 13:52:20.770714 1171 raft_consensus.cc:738] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.763254 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1273) took 4849 ms. Trace:
I20250625 13:52:20.771015 1168 rpcz_store.cc:276] 0625 13:52:15.914073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:15.914161 (+ 88us) service_pool.cc:225] Handling call
0625 13:52:20.701643 (+4787482us) ts_tablet_manager.cc:1145] Deleting tablet 2de9656958164dc4ae2fb0078b38f6c3
0625 13:52:20.701662 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.706106 (+ 4444us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.746936 (+ 40830us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.763234 (+ 16298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4787227}
I20250625 13:52:20.746538 2638 raft_consensus.cc:491] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.774645 2638 raft_consensus.cc:513] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.775291 2638 leader_election.cc:290] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.761082 1170 tablet_bootstrap.cc:492] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:20.772270 985 consensus_peers.cc:487] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a2007e51d424053bc2836dedf46add2. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:20.775767 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8985f259eb54ffdab29a9f6a2c5ae16" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.775964 915 raft_consensus.cc:2466] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:20.776427 983 leader_election.cc:304] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:20.776556 985 leader_election.cc:343] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d8985f259eb54ffdab29a9f6a2c5ae16
I20250625 13:52:20.776773 2638 raft_consensus.cc:2802] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:20.776844 2638 raft_consensus.cc:491] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:20.776875 1170 ts_tablet_manager.cc:1397] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.005s
I20250625 13:52:20.776906 2638 raft_consensus.cc:3058] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.777643 1170 raft_consensus.cc:357] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.777776 1170 raft_consensus.cc:383] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.777824 1170 raft_consensus.cc:738] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.778019 1170 consensus_queue.cc:260] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.779563 2638 raft_consensus.cc:513] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.779778 1170 ts_tablet_manager.cc:1428] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:20.779923 2254 log.cc:1199] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/b7d888e628a94f939ecf2571e60b1948
I20250625 13:52:20.780344 2254 ts_tablet_manager.cc:1939] T b7d888e628a94f939ecf2571e60b1948 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:20.781004 985 leader_election.cc:343] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d8985f259eb54ffdab29a9f6a2c5ae16
I20250625 13:52:20.781504 1171 consensus_queue.cc:260] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.781760 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8985f259eb54ffdab29a9f6a2c5ae16" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:20.781921 915 raft_consensus.cc:3058] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.782083 1171 ts_tablet_manager.cc:1428] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250625 13:52:20.784547 915 raft_consensus.cc:2466] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:20.785039 983 leader_election.cc:304] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.785295 2659 raft_consensus.cc:2802] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:20.785391 2659 raft_consensus.cc:695] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:20.785604 2659 consensus_queue.cc:237] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.786325 1168 tablet_replica.cc:331] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.786559 1168 raft_consensus.cc:2241] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.786722 1168 raft_consensus.cc:2270] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.786887 2594 raft_consensus.cc:491] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.787086 2594 raft_consensus.cc:513] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.788159 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bc97a56aaf741648bae711cffacc6fb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:20.788175 1168 ts_tablet_manager.cc:1905] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:20.788661 711 leader_election.cc:343] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.789407 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bc97a56aaf741648bae711cffacc6fb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:20.789902 711 leader_election.cc:343] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.790001 711 leader_election.cc:304] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:20.790237 2693 raft_consensus.cc:2747] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.790522 2638 leader_election.cc:290] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.790886 2594 leader_election.cc:290] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.791973 1171 tablet_bootstrap.cc:492] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.794261 1171 tablet_bootstrap.cc:654] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.795528 1170 tablet_bootstrap.cc:492] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.797726 1170 tablet_bootstrap.cc:654] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.800020 1166 tablet_bootstrap.cc:492] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:20.801623 1166 tablet_bootstrap.cc:654] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.801872 2638 raft_consensus.cc:491] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.803027 2638 raft_consensus.cc:513] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.771422 1165 ts_tablet_manager.cc:1918] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.803659 1165 log.cc:1199] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/4c9efc7631df4ad29112a41f229ae2e9
W20250625 13:52:20.804510 985 leader_election.cc:343] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f3c75deb64c74f0282efeed827659ec2
I20250625 13:52:20.804603 2638 leader_election.cc:290] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.804859 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c75deb64c74f0282efeed827659ec2" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.805110 915 raft_consensus.cc:2466] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:20.805557 983 leader_election.cc:304] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.806167 2638 raft_consensus.cc:2802] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:20.806279 2638 raft_consensus.cc:491] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:20.806278 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1349) took 2000 ms. Trace:
I20250625 13:52:20.806485 2254 rpcz_store.cc:276] 0625 13:52:18.805609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.809338 (+ 3729us) service_pool.cc:225] Handling call
0625 13:52:20.695741 (+1886403us) ts_tablet_manager.cc:1145] Deleting tablet b7d888e628a94f939ecf2571e60b1948
0625 13:52:20.707342 (+ 11601us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.735728 (+ 28386us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.738702 (+ 2974us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.806262 (+ 67560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885984}
I20250625 13:52:20.806869 1165 ts_tablet_manager.cc:1939] T 4c9efc7631df4ad29112a41f229ae2e9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:20.807497 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1350) took 1984 ms. Trace:
I20250625 13:52:20.807633 2254 rpcz_store.cc:276] 0625 13:52:18.822835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.837299 (+ 14464us) service_pool.cc:225] Handling call
0625 13:52:20.807334 (+1970035us) ts_tablet_manager.cc:1145] Deleting tablet 7e4fbede3f9040709ae4bdc9bbb792ad
0625 13:52:20.807343 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.807479 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969825}
I20250625 13:52:20.806406 2638 raft_consensus.cc:3058] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.810988 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1351) took 1963 ms. Trace:
I20250625 13:52:20.811108 2254 rpcz_store.cc:276] 0625 13:52:18.847928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.852302 (+ 4374us) service_pool.cc:225] Handling call
0625 13:52:20.810865 (+1958563us) ts_tablet_manager.cc:1145] Deleting tablet f0a59da2129141a698692c4b31609d1d
0625 13:52:20.810875 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.810973 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958328}
I20250625 13:52:20.811718 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1353) took 1943 ms. Trace:
I20250625 13:52:20.811877 2254 rpcz_store.cc:276] 0625 13:52:18.868380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.868460 (+ 80us) service_pool.cc:225] Handling call
0625 13:52:20.811590 (+1943130us) ts_tablet_manager.cc:1145] Deleting tablet a89db9678dd444d5a4df85baccd62044
0625 13:52:20.811601 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.811703 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942900}
I20250625 13:52:20.812366 2254 tablet_replica.cc:331] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:20.812382 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1434) took 1081 ms. Trace:
I20250625 13:52:20.812669 2254 raft_consensus.cc:2241] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.812880 2254 raft_consensus.cc:2270] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.813462 2254 ts_tablet_manager.cc:1905] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.810603 2638 raft_consensus.cc:513] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.815126 2638 leader_election.cc:290] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:20.818456 985 leader_election.cc:343] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: f3c75deb64c74f0282efeed827659ec2
I20250625 13:52:20.820950 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c75deb64c74f0282efeed827659ec2" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:20.821194 915 raft_consensus.cc:3058] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.823982 915 raft_consensus.cc:2466] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:20.824431 983 leader_election.cc:304] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.827353 1045 raft_consensus.cc:3058] T 423ca098138c476f861a19b44554fd24 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.827392 2638 raft_consensus.cc:2802] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:20.827800 2638 raft_consensus.cc:695] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:20.828037 2638 consensus_queue.cc:237] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.831660 1168 ts_tablet_manager.cc:1918] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:20.832197 1168 log.cc:1199] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/a30a35e3639a43c5b078169060824723
I20250625 13:52:20.835374 1165 rpcz_store.cc:276] 0625 13:52:19.731262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.039621 (+308359us) service_pool.cc:225] Handling call
0625 13:52:20.691828 (+652207us) ts_tablet_manager.cc:1145] Deleting tablet 4c9efc7631df4ad29112a41f229ae2e9
0625 13:52:20.691837 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.699295 (+ 7458us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.771269 (+ 71974us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.812361 (+ 41092us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":620210}
I20250625 13:52:20.837381 1165 tablet_replica.cc:331] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:20.837536 1166 tablet_bootstrap.cc:492] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:20.837630 1165 raft_consensus.cc:2241] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.837896 1165 raft_consensus.cc:2270] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.838511 1165 ts_tablet_manager.cc:1905] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.840346 1170 tablet_bootstrap.cc:492] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.840373 1168 ts_tablet_manager.cc:1939] T a30a35e3639a43c5b078169060824723 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:20.840596 1170 ts_tablet_manager.cc:1397] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.001s
I20250625 13:52:20.841333 1170 raft_consensus.cc:357] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.841598 1170 raft_consensus.cc:383] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.841764 1170 raft_consensus.cc:738] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.842073 1170 consensus_queue.cc:260] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.837733 1166 ts_tablet_manager.cc:1397] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250625 13:52:20.842645 1170 ts_tablet_manager.cc:1428] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:20.843071 1170 tablet_bootstrap.cc:492] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.843139 1166 raft_consensus.cc:357] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.843389 1166 raft_consensus.cc:383] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.843513 1166 raft_consensus.cc:738] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.843803 1166 consensus_queue.cc:260] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.846452 1171 tablet_bootstrap.cc:492] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.846627 1171 ts_tablet_manager.cc:1397] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.003s
I20250625 13:52:20.847391 1171 raft_consensus.cc:357] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.847632 1171 raft_consensus.cc:383] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.847743 1171 raft_consensus.cc:738] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.848024 1171 consensus_queue.cc:260] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.848538 1171 ts_tablet_manager.cc:1428] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:20.850023 1170 tablet_bootstrap.cc:654] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:20.856072 849 consensus_peers.cc:487] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:20.855037 1045 raft_consensus.cc:3058] T dec08d20a09444f5b9a102a8712e36c7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.859833 1171 tablet_bootstrap.cc:492] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.859825 1044 raft_consensus.cc:3058] T ef2b08fba37f4a38880c018ec2d869c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:20.861537 1171 tablet_bootstrap.cc:654] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.862746 1166 ts_tablet_manager.cc:1428] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250625 13:52:20.869671 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1276) took 4812 ms. Trace:
I20250625 13:52:20.869854 1168 rpcz_store.cc:276] 0625 13:52:16.056882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.060148 (+ 3266us) service_pool.cc:225] Handling call
0625 13:52:20.771437 (+4711289us) ts_tablet_manager.cc:1145] Deleting tablet a30a35e3639a43c5b078169060824723
0625 13:52:20.786269 (+ 14832us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.804141 (+ 17872us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.831494 (+ 27353us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.869649 (+ 38155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4711037}
I20250625 13:52:20.870429 1168 tablet_replica.cc:331] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.870664 1168 raft_consensus.cc:2241] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.871830 1166 tablet_bootstrap.cc:492] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:20.874809 1168 raft_consensus.cc:2270] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.875135 1166 tablet_bootstrap.cc:654] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.875481 1168 ts_tablet_manager.cc:1905] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.858450 2254 ts_tablet_manager.cc:1918] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:20.878360 2254 log.cc:1199] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/653267b468ef4958b98af0ed6a7bf4d3
I20250625 13:52:20.878845 2254 ts_tablet_manager.cc:1939] T 653267b468ef4958b98af0ed6a7bf4d3 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:20.881471 2667 raft_consensus.cc:491] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.881603 2667 raft_consensus.cc:513] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.883268 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e3ef7f5b474466bbc19fd31c6b6d99c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.884042 849 leader_election.cc:343] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.888674 2667 leader_election.cc:290] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:20.890041 851 leader_election.cc:343] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 0e3ef7f5b474466bbc19fd31c6b6d99c
I20250625 13:52:20.890166 851 leader_election.cc:304] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.890534 2667 raft_consensus.cc:2747] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.901464 1168 ts_tablet_manager.cc:1918] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:20.904245 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1352) took 2036 ms. Trace:
I20250625 13:52:20.905366 1165 ts_tablet_manager.cc:1918] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 13:52:20.905586 2254 rpcz_store.cc:276] 0625 13:52:18.868250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.868773 (+ 523us) service_pool.cc:225] Handling call
0625 13:52:20.812323 (+1943550us) ts_tablet_manager.cc:1145] Deleting tablet 653267b468ef4958b98af0ed6a7bf4d3
0625 13:52:20.812330 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.832625 (+ 20295us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.858342 (+ 25717us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.904221 (+ 45879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1943367}
I20250625 13:52:20.906201 2254 tablet_replica.cc:331] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:20.906482 2254 raft_consensus.cc:2241] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:20.906842 2254 pending_rounds.cc:70] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:20.907014 1165 log.cc:1199] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/50786a4c25274ffb8f9910c5dafe3933
I20250625 13:52:20.910585 1165 ts_tablet_manager.cc:1939] T 50786a4c25274ffb8f9910c5dafe3933 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:20.910130 1170 tablet_bootstrap.cc:492] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.910863 1170 ts_tablet_manager.cc:1397] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.004s
I20250625 13:52:20.911736 1170 raft_consensus.cc:357] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.911974 1170 raft_consensus.cc:383] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.912068 1170 raft_consensus.cc:738] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.912318 1170 consensus_queue.cc:260] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.914697 1170 ts_tablet_manager.cc:1428] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:20.915019 1170 tablet_bootstrap.cc:492] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.916744 1170 tablet_bootstrap.cc:654] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.922516 2254 pending_rounds.cc:77] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520675477770240 op_type: NO_OP noop_request { }
I20250625 13:52:20.922693 2254 raft_consensus.cc:2887] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:20.922870 2254 raft_consensus.cc:2270] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.921970 1166 tablet_bootstrap.cc:492] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:20.924144 1166 ts_tablet_manager.cc:1397] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.003s
I20250625 13:52:20.924904 1166 raft_consensus.cc:357] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.925222 1166 raft_consensus.cc:383] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.925323 1166 raft_consensus.cc:738] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.925649 1166 consensus_queue.cc:260] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.903887 1171 tablet_bootstrap.cc:492] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.928050 1171 ts_tablet_manager.cc:1397] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.001s
I20250625 13:52:20.928917 1171 raft_consensus.cc:357] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.929136 1171 raft_consensus.cc:383] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.929232 1171 raft_consensus.cc:738] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.929502 1171 consensus_queue.cc:260] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.930159 1171 ts_tablet_manager.cc:1428] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:20.930467 1171 tablet_bootstrap.cc:492] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.932811 1171 tablet_bootstrap.cc:654] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.935770 1166 ts_tablet_manager.cc:1428] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20250625 13:52:20.942637 1166 tablet_bootstrap.cc:492] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:20.944368 1166 tablet_bootstrap.cc:654] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.945353 1168 log.cc:1199] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:52:20.945840 1168 ts_tablet_manager.cc:1939] T c605d5b64c164c1594b0a40cfa2c4cf6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:20.945981 1170 tablet_bootstrap.cc:492] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.946132 1170 ts_tablet_manager.cc:1397] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250625 13:52:20.946890 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1277) took 4813 ms. Trace:
I20250625 13:52:20.946902 1170 raft_consensus.cc:357] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.947067 1170 raft_consensus.cc:383] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.947126 1170 raft_consensus.cc:738] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.947018 1168 rpcz_store.cc:276] 0625 13:52:16.133588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.133794 (+ 206us) service_pool.cc:225] Handling call
0625 13:52:20.870376 (+4736582us) ts_tablet_manager.cc:1145] Deleting tablet c605d5b64c164c1594b0a40cfa2c4cf6
0625 13:52:20.870386 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.897600 (+ 27214us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.901357 (+ 3757us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.946870 (+ 45513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4735449}
I20250625 13:52:20.947373 1170 consensus_queue.cc:260] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.947902 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1278) took 4707 ms. Trace:
I20250625 13:52:20.947943 1170 ts_tablet_manager.cc:1428] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:20.948025 1168 rpcz_store.cc:276] 0625 13:52:16.240068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.240125 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:20.947713 (+4707588us) ts_tablet_manager.cc:1145] Deleting tablet 5d467dbf133147e7a517c6fc4dff433a
0625 13:52:20.947722 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.947886 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4707409}
I20250625 13:52:20.948458 1168 tablet_replica.cc:331] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:20.948688 1168 raft_consensus.cc:2241] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:20.949065 1168 raft_consensus.cc:2270] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.952481 2254 ts_tablet_manager.cc:1905] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.955603 1170 tablet_bootstrap.cc:492] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:20.957834 1170 tablet_bootstrap.cc:654] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.959471 2638 raft_consensus.cc:491] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.959604 2638 raft_consensus.cc:513] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.960206 2638 leader_election.cc:290] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:20.960878 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:20.962561 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43474 (request call id 1437) took 1074 ms. Trace:
I20250625 13:52:20.962700 1165 rpcz_store.cc:276] 0625 13:52:19.888280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.065498 (+177218us) service_pool.cc:225] Handling call
0625 13:52:20.962468 (+896970us) ts_tablet_manager.cc:1145] Deleting tablet 0cf35321b38a4e81b41563f711dbb86d
0625 13:52:20.962476 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.962543 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":831755}
I20250625 13:52:20.963677 1165 tablet_replica.cc:331] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: stopping tablet replica
I20250625 13:52:20.963948 1165 raft_consensus.cc:2241] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:20.952438 2667 raft_consensus.cc:491] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:20.964186 1165 raft_consensus.cc:2270] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:20.964258 2667 raft_consensus.cc:513] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:20.964704 1165 ts_tablet_manager.cc:1905] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.964847 2667 leader_election.cc:290] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:20.966054 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db6e9db2375144b38ec43ffca9ca4c1e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:20.966536 849 leader_election.cc:343] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:20.962011 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38770320fc6541c39e10448978f8a3b0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:20.964129 1171 tablet_bootstrap.cc:492] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:20.967346 1171 ts_tablet_manager.cc:1397] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.003s
I20250625 13:52:20.967348 983 leader_election.cc:304] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:20.967984 2638 raft_consensus.cc:2747] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.968187 1171 raft_consensus.cc:357] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.968341 1171 raft_consensus.cc:383] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.968483 1171 raft_consensus.cc:738] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:52:20.968568 851 leader_election.cc:343] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: db6e9db2375144b38ec43ffca9ca4c1e
I20250625 13:52:20.968755 851 leader_election.cc:304] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:20.968791 1171 consensus_queue.cc:260] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.969022 2631 raft_consensus.cc:2747] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:20.977188 1171 ts_tablet_manager.cc:1428] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250625 13:52:20.977672 1171 tablet_bootstrap.cc:492] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:20.979408 1171 tablet_bootstrap.cc:654] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:20.983346 1170 tablet_bootstrap.cc:492] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:20.983577 1170 ts_tablet_manager.cc:1397] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250625 13:52:20.983850 1168 ts_tablet_manager.cc:1905] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:20.984419 1170 raft_consensus.cc:357] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.987358 1170 raft_consensus.cc:383] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:20.987473 1170 raft_consensus.cc:738] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:20.987767 1170 consensus_queue.cc:260] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:20.993140 1170 ts_tablet_manager.cc:1428] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250625 13:52:20.993891 1170 tablet_bootstrap.cc:492] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.000957 1170 tablet_bootstrap.cc:654] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.001948 1166 tablet_bootstrap.cc:492] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.002174 1166 ts_tablet_manager.cc:1397] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20250625 13:52:21.003264 1166 raft_consensus.cc:357] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.003576 1166 raft_consensus.cc:383] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.003682 1166 raft_consensus.cc:738] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.004019 1166 consensus_queue.cc:260] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.004582 1166 ts_tablet_manager.cc:1428] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:20.998735 2254 ts_tablet_manager.cc:1918] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:21.006249 2254 log.cc:1199] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/8e2bc4df39674dc3be0cd32c68f1ebca
I20250625 13:52:21.006718 2254 ts_tablet_manager.cc:1939] T 8e2bc4df39674dc3be0cd32c68f1ebca P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.007689 1166 tablet_bootstrap.cc:492] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.009397 1166 tablet_bootstrap.cc:654] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.024914 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1372) took 1938 ms. Trace:
I20250625 13:52:21.026432 1165 ts_tablet_manager.cc:1918] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:21.025045 2254 rpcz_store.cc:276] 0625 13:52:19.086967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.087025 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:20.906148 (+1819123us) ts_tablet_manager.cc:1145] Deleting tablet 8e2bc4df39674dc3be0cd32c68f1ebca
0625 13:52:20.906159 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.977850 (+ 71691us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:20.998632 (+ 20782us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.024886 (+ 26254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818861}
I20250625 13:52:21.027771 1165 log.cc:1199] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-0/wal/wals/4c49ef682cbb46e1ae5c3b91e4e2d9c9
I20250625 13:52:21.028254 1165 ts_tablet_manager.cc:1939] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 5c1aa07ab6bc4b0089e46951240a37c9: Deleting consensus metadata
I20250625 13:52:21.031405 2254 tablet_replica.cc:331] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.031694 2254 raft_consensus.cc:2241] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:21.032074 2254 pending_rounds.cc:70] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:21.032204 2254 pending_rounds.cc:77] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520675650494464 op_type: NO_OP noop_request { }
I20250625 13:52:21.032382 2254 raft_consensus.cc:2887] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:21.032527 2254 raft_consensus.cc:2270] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.033111 1170 tablet_bootstrap.cc:492] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.033283 1170 ts_tablet_manager.cc:1397] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250625 13:52:21.033996 1166 tablet_bootstrap.cc:492] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.033928 1170 raft_consensus.cc:357] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.034065 1170 raft_consensus.cc:383] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.034173 1170 raft_consensus.cc:738] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.034219 1166 ts_tablet_manager.cc:1397] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20250625 13:52:21.034757 1170 consensus_queue.cc:260] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.035018 1166 raft_consensus.cc:357] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.042459 1170 ts_tablet_manager.cc:1428] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250625 13:52:21.042851 1170 tablet_bootstrap.cc:492] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.042886 2631 raft_consensus.cc:491] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.043033 2631 raft_consensus.cc:513] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.043675 2631 leader_election.cc:290] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:21.045822 851 leader_election.cc:343] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 32a57268a18f4f00a39f8360f9c90737
I20250625 13:52:21.046792 1171 tablet_bootstrap.cc:492] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.047057 1171 ts_tablet_manager.cc:1397] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.000s
I20250625 13:52:21.044669 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a57268a18f4f00a39f8360f9c90737" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.047901 1171 raft_consensus.cc:357] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.048048 1171 raft_consensus.cc:383] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:52:21.048058 849 leader_election.cc:343] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.048144 1171 raft_consensus.cc:738] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.048206 849 leader_election.cc:304] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.048410 2631 raft_consensus.cc:2747] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.048568 1171 consensus_queue.cc:260] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.049256 1171 ts_tablet_manager.cc:1428] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:21.050446 1170 tablet_bootstrap.cc:654] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.053006 1171 tablet_bootstrap.cc:492] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.033142 2254 ts_tablet_manager.cc:1905] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 13:52:21.053668 985 consensus_peers.cc:487] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 077e8c001cc646559b5268d4a48c0ad6. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:21.054693 2638 raft_consensus.cc:491] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.054786 1171 tablet_bootstrap.cc:654] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.054812 2638 raft_consensus.cc:513] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.055358 2638 leader_election.cc:290] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.056174 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9462da7574a46839763b1c388102e60" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.056365 915 raft_consensus.cc:2466] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:21.057197 983 leader_election.cc:304] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.057509 2638 raft_consensus.cc:2802] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.057586 2638 raft_consensus.cc:491] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.057643 2638 raft_consensus.cc:3058] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.059386 1166 raft_consensus.cc:383] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.059478 1166 raft_consensus.cc:738] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.059744 1166 consensus_queue.cc:260] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.060317 1166 ts_tablet_manager.cc:1428] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250625 13:52:21.061386 1168 ts_tablet_manager.cc:1918] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:21.061554 1168 log.cc:1199] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/ab7202a8761444a0a731993d890fcdeb
I20250625 13:52:21.061736 2638 raft_consensus.cc:513] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.062084 1168 ts_tablet_manager.cc:1939] T ab7202a8761444a0a731993d890fcdeb P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.062717 2631 raft_consensus.cc:491] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.062852 2631 raft_consensus.cc:513] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.063293 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9462da7574a46839763b1c388102e60" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.063467 2631 leader_election.cc:290] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.063498 2638 leader_election.cc:290] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.063998 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ae5f1d414794474b2178b431cbc1d44" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.064471 849 leader_election.cc:343] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.063465 915 raft_consensus.cc:3058] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.067744 915 raft_consensus.cc:2466] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.068380 983 leader_election.cc:304] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.068703 2638 raft_consensus.cc:2802] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.068887 2638 raft_consensus.cc:695] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:21.069162 2638 consensus_queue.cc:237] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:21.077629 985 leader_election.cc:343] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d9462da7574a46839763b1c388102e60
W20250625 13:52:21.079062 985 leader_election.cc:343] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d9462da7574a46839763b1c388102e60
I20250625 13:52:21.079465 1166 tablet_bootstrap.cc:492] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:21.079856 851 leader_election.cc:343] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4ae5f1d414794474b2178b431cbc1d44
I20250625 13:52:21.079965 851 leader_election.cc:304] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.080406 2573 raft_consensus.cc:2747] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.081143 1166 tablet_bootstrap.cc:654] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.088111 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1281) took 4741 ms. Trace:
I20250625 13:52:21.090173 1168 rpcz_store.cc:276] 0625 13:52:16.346968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.347019 (+ 51us) service_pool.cc:225] Handling call
0625 13:52:20.948418 (+4601399us) ts_tablet_manager.cc:1145] Deleting tablet ab7202a8761444a0a731993d890fcdeb
0625 13:52:20.948426 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:20.986962 (+ 38536us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.061272 (+ 74310us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.088086 (+ 26814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4601241}
I20250625 13:52:21.089661 1171 tablet_bootstrap.cc:492] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.094565 1171 ts_tablet_manager.cc:1397] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.003s
I20250625 13:52:21.095402 1171 raft_consensus.cc:357] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.095631 1171 raft_consensus.cc:383] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.095711 1171 raft_consensus.cc:738] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.095923 1171 consensus_queue.cc:260] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.096424 1171 ts_tablet_manager.cc:1428] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.099911 1170 tablet_bootstrap.cc:492] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.100111 1170 ts_tablet_manager.cc:1397] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.004s
I20250625 13:52:21.101092 1170 raft_consensus.cc:357] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.101315 1170 raft_consensus.cc:383] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.101418 1170 raft_consensus.cc:738] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.101677 1170 consensus_queue.cc:260] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.102180 1170 ts_tablet_manager.cc:1428] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:21.102506 2254 ts_tablet_manager.cc:1918] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:21.103461 2254 log.cc:1199] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/9a2007e51d424053bc2836dedf46add2
I20250625 13:52:21.107714 2254 ts_tablet_manager.cc:1939] T 9a2007e51d424053bc2836dedf46add2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.108501 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1373) took 1975 ms. Trace:
I20250625 13:52:21.108664 2254 rpcz_store.cc:276] 0625 13:52:19.132735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.151387 (+ 18652us) service_pool.cc:225] Handling call
0625 13:52:21.027689 (+1876302us) ts_tablet_manager.cc:1145] Deleting tablet 9a2007e51d424053bc2836dedf46add2
0625 13:52:21.031347 (+ 3658us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.067583 (+ 36236us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.102402 (+ 34819us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.108482 (+ 6080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874757}
I20250625 13:52:21.102810 1170 tablet_bootstrap.cc:492] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.111297 1170 tablet_bootstrap.cc:654] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.113368 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1374) took 1791 ms. Trace:
I20250625 13:52:21.113538 2254 rpcz_store.cc:276] 0625 13:52:19.322208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.322274 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:21.113214 (+1790940us) ts_tablet_manager.cc:1145] Deleting tablet 7e8396ea40d34c66ab740c66647a6afc
0625 13:52:21.113225 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.113347 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1780873}
I20250625 13:52:21.114184 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1375) took 1762 ms. Trace:
I20250625 13:52:21.114338 2254 rpcz_store.cc:276] 0625 13:52:19.351392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.351458 (+ 66us) service_pool.cc:225] Handling call
0625 13:52:21.114070 (+1762612us) ts_tablet_manager.cc:1145] Deleting tablet cbd82bb415174dbdbcfafa9d6c3f3aa7
0625 13:52:21.114081 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.114168 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1762381}
I20250625 13:52:21.114979 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1376) took 1762 ms. Trace:
I20250625 13:52:21.115090 2254 rpcz_store.cc:276] 0625 13:52:19.352329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.352435 (+ 106us) service_pool.cc:225] Handling call
0625 13:52:21.114825 (+1762390us) ts_tablet_manager.cc:1145] Deleting tablet 7e8396ea40d34c66ab740c66647a6afc
0625 13:52:21.114834 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.114900 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1762232}
I20250625 13:52:21.115526 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1377) took 1740 ms. Trace:
I20250625 13:52:21.115640 2254 rpcz_store.cc:276] 0625 13:52:19.374733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.374785 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:21.115446 (+1740661us) ts_tablet_manager.cc:1145] Deleting tablet d8257f79a42247b48f2736a545362f44
0625 13:52:21.115454 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.115514 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1740494}
I20250625 13:52:21.116123 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1385) took 1683 ms. Trace:
I20250625 13:52:21.116268 2254 rpcz_store.cc:276] 0625 13:52:19.432403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.432447 (+ 44us) service_pool.cc:225] Handling call
0625 13:52:21.116014 (+1683567us) ts_tablet_manager.cc:1145] Deleting tablet e1549b9e797a44f89b316a44c5e7ee2e
0625 13:52:21.116026 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.116106 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1683403}
I20250625 13:52:21.115778 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1282) took 4762 ms. Trace:
I20250625 13:52:21.117094 1168 rpcz_store.cc:276] 0625 13:52:16.353656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.353724 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:21.094420 (+4740696us) ts_tablet_manager.cc:1145] Deleting tablet 23fa1b7e7c13444fb8ee07807dabd813
0625 13:52:21.115615 (+ 21195us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.115751 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4740529}
I20250625 13:52:21.117432 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1386) took 1683 ms. Trace:
I20250625 13:52:21.117587 2254 rpcz_store.cc:276] 0625 13:52:19.434262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.434350 (+ 88us) service_pool.cc:225] Handling call
0625 13:52:21.117347 (+1682997us) ts_tablet_manager.cc:1145] Deleting tablet e58128254acf44129f7b4e452ac03fe5
0625 13:52:21.117355 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.117418 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682812}
I20250625 13:52:21.119006 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1283) took 4697 ms. Trace:
I20250625 13:52:21.119099 1168 rpcz_store.cc:276] 0625 13:52:16.419535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.419791 (+ 256us) service_pool.cc:225] Handling call
0625 13:52:21.117338 (+4697547us) ts_tablet_manager.cc:1145] Deleting tablet 882152a3a5a2458d803ba7f75b31bc30
0625 13:52:21.117347 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.117437 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4684787}
I20250625 13:52:21.119235 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1387) took 1666 ms. Trace:
I20250625 13:52:21.119446 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1286) took 4648 ms. Trace:
I20250625 13:52:21.119426 2254 rpcz_store.cc:276] 0625 13:52:19.453051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.458181 (+ 5130us) service_pool.cc:225] Handling call
0625 13:52:21.119146 (+1660965us) ts_tablet_manager.cc:1145] Deleting tablet fc086d416d0f475987bd67f122487353
0625 13:52:21.119155 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.119220 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660735}
I20250625 13:52:21.119884 2254 tablet_replica.cc:331] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.120111 2254 raft_consensus.cc:2241] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:21.119551 1168 rpcz_store.cc:276] 0625 13:52:16.470674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:16.470726 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:21.119337 (+4648611us) ts_tablet_manager.cc:1145] Deleting tablet e38a85c299f04d60a3c465571fd33971
0625 13:52:21.119347 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.119430 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4648446}
I20250625 13:52:21.120834 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1291) took 4010 ms. Trace:
I20250625 13:52:21.120965 2254 pending_rounds.cc:70] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:21.120922 1168 rpcz_store.cc:276] 0625 13:52:17.110228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.110276 (+ 48us) service_pool.cc:225] Handling call
0625 13:52:21.120712 (+4010436us) ts_tablet_manager.cc:1145] Deleting tablet 54cc2d0cf95b41d6b36483af5436fcde
0625 13:52:21.120721 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.120822 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4009438}
I20250625 13:52:21.121102 2254 pending_rounds.cc:77] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520676469379072 op_type: NO_OP noop_request { }
I20250625 13:52:21.121213 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1292) took 3998 ms. Trace:
I20250625 13:52:21.119727 2728 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d85834f366fa42c294c9290af21ea111)
I20250625 13:52:21.121485 2254 raft_consensus.cc:2887] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:21.121524 2728 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:21.122660 2254 raft_consensus.cc:2270] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.122758 1168 rpcz_store.cc:276] 0625 13:52:17.122719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.122888 (+ 169us) service_pool.cc:225] Handling call
0625 13:52:21.121111 (+3998223us) ts_tablet_manager.cc:1145] Deleting tablet 575043022b1b4ca686775b2aae41a037
0625 13:52:21.121119 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.121196 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3997194}
I20250625 13:52:21.123064 2732 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d85834f366fa42c294c9290af21ea111)
I20250625 13:52:21.122607 1171 tablet_bootstrap.cc:492] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.124024 1170 tablet_bootstrap.cc:492] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.124189 1170 ts_tablet_manager.cc:1397] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20250625 13:52:21.124943 2638 raft_consensus.cc:491] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.125026 1171 tablet_bootstrap.cc:654] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.125128 2638 raft_consensus.cc:513] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.125754 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1295) took 3936 ms. Trace:
I20250625 13:52:21.125830 1168 rpcz_store.cc:276] 0625 13:52:17.189045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.189102 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:21.125647 (+3936545us) ts_tablet_manager.cc:1145] Deleting tablet 6352e917532840ca832e32ddf0747da4
0625 13:52:21.125654 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.125740 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3936337}
I20250625 13:52:21.126561 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1296) took 3922 ms. Trace:
W20250625 13:52:21.126570 985 leader_election.cc:343] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d26820eaba98401da3a8b5348be21fc6
I20250625 13:52:21.126626 1168 rpcz_store.cc:276] 0625 13:52:17.203886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.207671 (+ 3785us) service_pool.cc:225] Handling call
0625 13:52:21.126460 (+3918789us) ts_tablet_manager.cc:1145] Deleting tablet a6b2f861d63d4299b3e472cf705a1b67
0625 13:52:21.126468 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.126549 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3918579}
I20250625 13:52:21.127341 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26820eaba98401da3a8b5348be21fc6" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.127400 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1300) took 3824 ms. Trace:
I20250625 13:52:21.127501 1168 rpcz_store.cc:276] 0625 13:52:17.302635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.305311 (+ 2676us) service_pool.cc:225] Handling call
0625 13:52:21.127246 (+3821935us) ts_tablet_manager.cc:1145] Deleting tablet aef43c4962c747dbb43039d1e706bbd9
0625 13:52:21.127255 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.127379 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3821711}
I20250625 13:52:21.127717 2638 leader_election.cc:290] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.127511 915 raft_consensus.cc:2466] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:21.128469 1168 tablet_replica.cc:331] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.128575 983 leader_election.cc:304] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.128695 1168 raft_consensus.cc:2241] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.128837 1168 raft_consensus.cc:2270] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.129019 2638 raft_consensus.cc:2802] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.129132 2638 raft_consensus.cc:491] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.129247 2638 raft_consensus.cc:3058] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.129401 1168 ts_tablet_manager.cc:1905] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.129482 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1849 } } ignore_live_leader: false dest_uuid: "58351267767f4d9a85f974ebc40db6fe" is_pre_election: true
I20250625 13:52:21.123282 2732 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:21.131975 2638 raft_consensus.cc:513] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.132638 535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fec9d8a79054a5991ae90d8a3101e32 in term 1.
W20250625 13:52:21.133338 985 leader_election.cc:343] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: d26820eaba98401da3a8b5348be21fc6
I20250625 13:52:21.133566 2728 leader_election.cc:290] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58351267767f4d9a85f974ebc40db6fe (127.0.104.61:45727), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
I20250625 13:52:21.133531 1170 raft_consensus.cc:357] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.133756 1170 raft_consensus.cc:383] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.133838 1170 raft_consensus.cc:738] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.134063 1170 consensus_queue.cc:260] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.135190 1997 leader_election.cc:304] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fec9d8a79054a5991ae90d8a3101e32, 58351267767f4d9a85f974ebc40db6fe; no voters:
I20250625 13:52:21.135725 2728 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 13:52:21.135844 2728 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d85834f366fa42c294c9290af21ea111)
I20250625 13:52:21.135937 2728 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 1 FOLLOWER]: Advancing to term 2
W20250625 13:52:21.138031 851 consensus_peers.cc:487] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c49ef682cbb46e1ae5c3b91e4e2d9c9. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:21.138500 2732 leader_election.cc:290] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
I20250625 13:52:21.138844 1170 ts_tablet_manager.cc:1428] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250625 13:52:21.139206 1170 tablet_bootstrap.cc:492] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.140187 2638 leader_election.cc:290] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.140895 2254 ts_tablet_manager.cc:1905] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.141495 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d26820eaba98401da3a8b5348be21fc6" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.141674 915 raft_consensus.cc:3058] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.144270 915 raft_consensus.cc:2466] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.144994 983 leader_election.cc:304] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.146425 1166 tablet_bootstrap.cc:492] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.146658 1166 ts_tablet_manager.cc:1397] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.006s
I20250625 13:52:21.147668 1166 raft_consensus.cc:357] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.142302 1170 tablet_bootstrap.cc:654] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.153501 1166 raft_consensus.cc:383] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.153625 1166 raft_consensus.cc:738] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.153860 1166 consensus_queue.cc:260] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.154758 1171 tablet_bootstrap.cc:492] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.159895 1166 ts_tablet_manager.cc:1428] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250625 13:52:21.160957 2728 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:21.172933 2594 raft_consensus.cc:491] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.173166 2594 raft_consensus.cc:513] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.173889 2594 leader_election.cc:290] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.174571 2728 leader_election.cc:290] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 election: Requested vote from peers 58351267767f4d9a85f974ebc40db6fe (127.0.104.61:45727), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
I20250625 13:52:21.175153 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a289b9e3fac4fd8bacc22c235f7010c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.173918 1166 tablet_bootstrap.cc:492] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.175559 2254 ts_tablet_manager.cc:1918] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250625 13:52:21.175796 711 leader_election.cc:343] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.176051 1171 ts_tablet_manager.cc:1397] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.001s
I20250625 13:52:21.175215 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a289b9e3fac4fd8bacc22c235f7010c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.176826 915 raft_consensus.cc:2466] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:21.161558 1168 ts_tablet_manager.cc:1918] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.177206 1168 log.cc:1199] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d8985f259eb54ffdab29a9f6a2c5ae16
I20250625 13:52:21.177314 711 leader_election.cc:304] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:21.177647 1168 ts_tablet_manager.cc:1939] T d8985f259eb54ffdab29a9f6a2c5ae16 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.178377 2594 raft_consensus.cc:2802] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.178457 2594 raft_consensus.cc:491] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.178529 2594 raft_consensus.cc:3058] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.178539 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1303) took 3836 ms. Trace:
I20250625 13:52:21.178644 1168 rpcz_store.cc:276] 0625 13:52:17.342338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.342501 (+ 163us) service_pool.cc:225] Handling call
0625 13:52:21.128418 (+3785917us) ts_tablet_manager.cc:1145] Deleting tablet d8985f259eb54ffdab29a9f6a2c5ae16
0625 13:52:21.128430 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.134665 (+ 6235us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.161424 (+ 26759us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.178519 (+ 17095us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3785715}
I20250625 13:52:21.179031 1168 tablet_replica.cc:331] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.179251 1168 raft_consensus.cc:2241] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.179451 1168 raft_consensus.cc:2270] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.179975 1168 ts_tablet_manager.cc:1905] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.181221 2594 raft_consensus.cc:513] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.183054 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a289b9e3fac4fd8bacc22c235f7010c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.183256 915 raft_consensus.cc:3058] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.184135 1171 raft_consensus.cc:357] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.185595 1171 raft_consensus.cc:383] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.185693 1171 raft_consensus.cc:738] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.185899 2594 leader_election.cc:290] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.185942 1171 consensus_queue.cc:260] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.177055 1166 tablet_bootstrap.cc:654] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.186367 2254 log.cc:1199] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/077e8c001cc646559b5268d4a48c0ad6
I20250625 13:52:21.186167 915 raft_consensus.cc:2466] T 5a289b9e3fac4fd8bacc22c235f7010c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:21.186822 711 leader_election.cc:304] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.187114 2594 raft_consensus.cc:2802] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.187626 2594 raft_consensus.cc:695] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:21.187985 2594 consensus_queue.cc:237] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.180899 2638 raft_consensus.cc:2802] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.182300 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a289b9e3fac4fd8bacc22c235f7010c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:21.189324 2254 ts_tablet_manager.cc:1939] T 077e8c001cc646559b5268d4a48c0ad6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
W20250625 13:52:21.189909 711 leader_election.cc:343] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.190042 2580 raft_consensus.cc:491] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.190147 2580 raft_consensus.cc:513] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.190719 2580 leader_election.cc:290] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.190698 2638 raft_consensus.cc:695] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:21.190960 2638 consensus_queue.cc:237] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.191124 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:21.191210 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.191288 764 raft_consensus.cc:2466] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:21.185488 1170 tablet_bootstrap.cc:492] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.191664 1170 ts_tablet_manager.cc:1397] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20250625 13:52:21.191709 851 leader_election.cc:304] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:21.191850 849 leader_election.cc:343] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.192127 2580 raft_consensus.cc:2802] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.192203 2580 raft_consensus.cc:491] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.192272 2580 raft_consensus.cc:3058] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.192430 1170 raft_consensus.cc:357] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.192582 1170 raft_consensus.cc:383] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.192637 1170 raft_consensus.cc:738] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.192852 1170 consensus_queue.cc:260] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.194777 2580 raft_consensus.cc:513] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.195117 1170 ts_tablet_manager.cc:1428] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:21.195418 1170 tablet_bootstrap.cc:492] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.195463 2580 leader_election.cc:290] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.196089 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:21.196254 764 raft_consensus.cc:3058] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.197001 1170 tablet_bootstrap.cc:654] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.198897 764 raft_consensus.cc:2466] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:21.199326 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:21.199998 851 leader_election.cc:304] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.201084 1170 tablet_bootstrap.cc:492] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.201205 1170 ts_tablet_manager.cc:1397] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:52:21.201442 2580 raft_consensus.cc:2802] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.201603 2580 raft_consensus.cc:695] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
W20250625 13:52:21.201988 849 leader_election.cc:343] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.202026 1170 raft_consensus.cc:357] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.202055 2580 consensus_queue.cc:237] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.202163 1170 raft_consensus.cc:383] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.202210 1170 raft_consensus.cc:738] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.202421 1170 consensus_queue.cc:260] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.202976 1170 ts_tablet_manager.cc:1428] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.203229 1170 tablet_bootstrap.cc:492] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.203817 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fec9d8a79054a5991ae90d8a3101e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1849 } } ignore_live_leader: false dest_uuid: "58351267767f4d9a85f974ebc40db6fe"
I20250625 13:52:21.204041 535 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:52:21.204892 1170 tablet_bootstrap.cc:654] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.206840 2693 raft_consensus.cc:491] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.206969 2693 raft_consensus.cc:513] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.207653 2693 leader_election.cc:290] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.208285 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.208760 1171 ts_tablet_manager.cc:1428] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
W20250625 13:52:21.209105 711 leader_election.cc:343] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.209182 1171 tablet_bootstrap.cc:492] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.209862 535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fec9d8a79054a5991ae90d8a3101e32 in term 2.
I20250625 13:52:21.210616 1171 tablet_bootstrap.cc:654] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.210590 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.211125 711 leader_election.cc:343] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.211232 711 leader_election.cc:304] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.211714 2693 raft_consensus.cc:2747] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.212546 1997 leader_election.cc:304] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fec9d8a79054a5991ae90d8a3101e32, 58351267767f4d9a85f974ebc40db6fe; no voters:
I20250625 13:52:21.212983 2728 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:52:21.213335 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1403) took 1598 ms. Trace:
I20250625 13:52:21.213528 2254 rpcz_store.cc:276] 0625 13:52:19.614402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.614580 (+ 178us) service_pool.cc:225] Handling call
0625 13:52:21.119843 (+1505263us) ts_tablet_manager.cc:1145] Deleting tablet 077e8c001cc646559b5268d4a48c0ad6
0625 13:52:21.119851 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.145260 (+ 25409us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.175417 (+ 30157us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.213300 (+ 37883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503994}
I20250625 13:52:21.214390 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1417) took 1486 ms. Trace:
I20250625 13:52:21.214536 2254 rpcz_store.cc:276] 0625 13:52:19.727432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.729441 (+ 2009us) service_pool.cc:225] Handling call
0625 13:52:21.214227 (+1484786us) ts_tablet_manager.cc:1145] Deleting tablet 3775c883f8534c6d90687152b9ca22d1
0625 13:52:21.214244 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.214371 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1483743}
I20250625 13:52:21.215127 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1418) took 1397 ms. Trace:
I20250625 13:52:21.215268 2254 rpcz_store.cc:276] 0625 13:52:19.818017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.818086 (+ 69us) service_pool.cc:225] Handling call
0625 13:52:21.215013 (+1396927us) ts_tablet_manager.cc:1145] Deleting tablet 0cf35321b38a4e81b41563f711dbb86d
0625 13:52:21.215024 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.215111 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1396695}
I20250625 13:52:21.215876 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1419) took 1395 ms. Trace:
I20250625 13:52:21.216007 2254 rpcz_store.cc:276] 0625 13:52:19.820881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.821076 (+ 195us) service_pool.cc:225] Handling call
0625 13:52:21.215780 (+1394704us) ts_tablet_manager.cc:1145] Deleting tablet 34fe02d7b37f4d99bbae082943223e35
0625 13:52:21.215791 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.215861 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394515}
I20250625 13:52:21.216668 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1421) took 1385 ms. Trace:
I20250625 13:52:21.217515 1166 tablet_bootstrap.cc:492] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.217698 1166 ts_tablet_manager.cc:1397] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.002s
I20250625 13:52:21.218604 1166 raft_consensus.cc:357] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.218828 1166 raft_consensus.cc:383] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.218919 1166 raft_consensus.cc:738] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.219210 1166 consensus_queue.cc:260] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.219934 1166 ts_tablet_manager.cc:1428] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:21.220351 1166 tablet_bootstrap.cc:492] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.222440 1166 tablet_bootstrap.cc:654] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.229435 2728 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [term 2 LEADER]: Becoming Leader. State: Replica: 3fec9d8a79054a5991ae90d8a3101e32, State: Running, Role: LEADER
I20250625 13:52:21.233228 2728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1848, Committed index: 1848, Last appended: 1.1849, Last appended by leader: 1849, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:21.235585 2744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fec9d8a79054a5991ae90d8a3101e32. Latest consensus state: current_term: 2 leader_uuid: "3fec9d8a79054a5991ae90d8a3101e32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:21.235879 2744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: LEADER
I20250625 13:52:21.240814 2091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58351267767f4d9a85f974ebc40db6fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1848 } } ignore_live_leader: false dest_uuid: "3fec9d8a79054a5991ae90d8a3101e32" is_pre_election: true
I20250625 13:52:21.241647 2254 rpcz_store.cc:276] 0625 13:52:19.831580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.831645 (+ 65us) service_pool.cc:225] Handling call
0625 13:52:21.216540 (+1384895us) ts_tablet_manager.cc:1145] Deleting tablet ba762a8edb8847368103e5bede9e9d0b
0625 13:52:21.216565 (+ 25us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.216638 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1384688}
I20250625 13:52:21.222641 2659 raft_consensus.cc:491] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.242091 2659 raft_consensus.cc:513] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.242229 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1420) took 1411 ms. Trace:
I20250625 13:52:21.242326 2254 rpcz_store.cc:276] 0625 13:52:19.830816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.845608 (+ 14792us) service_pool.cc:225] Handling call
0625 13:52:21.242076 (+1396468us) ts_tablet_manager.cc:1145] Deleting tablet 50ac647b1d704142b94da8d506319466
0625 13:52:21.242087 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.242183 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322452}
I20250625 13:52:21.242712 2659 leader_election.cc:290] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.222988 2636 raft_consensus.cc:491] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.242888 2636 raft_consensus.cc:513] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.243186 2254 tablet_replica.cc:331] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.243415 2254 raft_consensus.cc:2241] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.237128 1170 tablet_bootstrap.cc:492] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:21.245417 985 leader_election.cc:343] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 829221b8a79f460d90dd24be6a2d4c43
W20250625 13:52:21.247661 985 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: da2f8d0d9a714a28ba6fb9642d461de2
I20250625 13:52:21.243412 2636 leader_election.cc:290] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.247964 1170 ts_tablet_manager.cc:1397] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20250625 13:52:21.229110 2746 raft_consensus.cc:491] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.248198 2746 raft_consensus.cc:513] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.248733 1170 raft_consensus.cc:357] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:21.248900 1170 raft_consensus.cc:383] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.248967 1170 raft_consensus.cc:738] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.249162 1170 consensus_queue.cc:260] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:21.249714 1170 ts_tablet_manager.cc:1428] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:21.249910 2254 raft_consensus.cc:2270] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.250357 2580 raft_consensus.cc:491] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.250460 2254 ts_tablet_manager.cc:1905] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.250499 2580 raft_consensus.cc:513] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.251464 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7efc8df3c8e24020b073a1b61bc9e0a0" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.251958 849 leader_election.cc:343] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.252308 2580 leader_election.cc:290] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.265002 1170 tablet_bootstrap.cc:492] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.267163 1170 tablet_bootstrap.cc:654] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:21.269716 985 leader_election.cc:343] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7506cce0ec4c4ec68f25591cb98b1252
W20250625 13:52:21.269739 851 leader_election.cc:343] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7efc8df3c8e24020b073a1b61bc9e0a0
I20250625 13:52:21.269868 851 leader_election.cc:304] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.270210 2580 raft_consensus.cc:2747] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.270269 2746 leader_election.cc:290] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.278199 1166 tablet_bootstrap.cc:492] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.278435 1166 ts_tablet_manager.cc:1397] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.001s
I20250625 13:52:21.279263 1166 raft_consensus.cc:357] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.279542 1166 raft_consensus.cc:383] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.279646 1166 raft_consensus.cc:738] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.279903 1166 consensus_queue.cc:260] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.280097 1170 tablet_bootstrap.cc:492] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.280470 1166 ts_tablet_manager.cc:1428] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:21.281198 1170 ts_tablet_manager.cc:1397] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250625 13:52:21.281912 1170 raft_consensus.cc:357] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.282332 2580 raft_consensus.cc:491] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.282452 2580 raft_consensus.cc:513] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.283010 2580 leader_election.cc:290] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.282145 1170 raft_consensus.cc:383] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.283663 1170 raft_consensus.cc:738] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.283629 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9be75d03121f4ea2906236ac3a5b0764" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.284013 1170 consensus_queue.cc:260] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.284716 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7506cce0ec4c4ec68f25591cb98b1252" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.284893 916 raft_consensus.cc:2466] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:21.285408 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829221b8a79f460d90dd24be6a2d4c43" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.285566 915 raft_consensus.cc:2466] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
W20250625 13:52:21.285893 849 leader_election.cc:343] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:21.286767 851 leader_election.cc:343] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9be75d03121f4ea2906236ac3a5b0764
I20250625 13:52:21.286798 983 leader_election.cc:304] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.286870 851 leader_election.cc:304] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
W20250625 13:52:21.287190 983 leader_election.cc:343] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: da2f8d0d9a714a28ba6fb9642d461de2
I20250625 13:52:21.287292 983 leader_election.cc:304] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.287547 2659 raft_consensus.cc:2802] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.287657 2659 raft_consensus.cc:491] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.287662 983 leader_election.cc:304] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.288079 2580 raft_consensus.cc:2747] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.288106 2638 raft_consensus.cc:2802] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.288192 2638 raft_consensus.cc:491] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.288214 1168 ts_tablet_manager.cc:1918] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.288273 2638 raft_consensus.cc:3058] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.288306 1168 log.cc:1199] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/077e8c001cc646559b5268d4a48c0ad6
I20250625 13:52:21.288589 1168 ts_tablet_manager.cc:1939] T 077e8c001cc646559b5268d4a48c0ad6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.288678 1170 ts_tablet_manager.cc:1428] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:21.288975 1170 tablet_bootstrap.cc:492] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.289345 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1304) took 3946 ms. Trace:
I20250625 13:52:21.289418 1168 rpcz_store.cc:276] 0625 13:52:17.342449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.342730 (+ 281us) service_pool.cc:225] Handling call
0625 13:52:21.178974 (+3836244us) ts_tablet_manager.cc:1145] Deleting tablet 077e8c001cc646559b5268d4a48c0ad6
0625 13:52:21.178987 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.218605 (+ 39618us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.288105 (+ 69500us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.289327 (+ 1222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3836026}
I20250625 13:52:21.290916 1170 tablet_bootstrap.cc:654] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.290952 2638 raft_consensus.cc:513] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.287741 2636 raft_consensus.cc:2747] T da2f8d0d9a714a28ba6fb9642d461de2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.287757 2659 raft_consensus.cc:3058] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.292066 1171 tablet_bootstrap.cc:492] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.292230 1171 ts_tablet_manager.cc:1397] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.003s
I20250625 13:52:21.292948 1171 raft_consensus.cc:357] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.293123 1171 raft_consensus.cc:383] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.293172 1171 raft_consensus.cc:738] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.293385 1171 consensus_queue.cc:260] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.294000 1171 ts_tablet_manager.cc:1428] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:21.294351 2659 raft_consensus.cc:513] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.295817 2659 leader_election.cc:290] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:21.296164 985 leader_election.cc:343] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7506cce0ec4c4ec68f25591cb98b1252
I20250625 13:52:21.296833 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7506cce0ec4c4ec68f25591cb98b1252" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.296988 915 raft_consensus.cc:3058] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.299173 2254 ts_tablet_manager.cc:1918] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 13:52:21.299296 2254 log.cc:1199] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/50786a4c25274ffb8f9910c5dafe3933
I20250625 13:52:21.299755 915 raft_consensus.cc:2466] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.300207 983 leader_election.cc:304] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.303721 1168 tablet_replica.cc:331] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.303081 2638 leader_election.cc:290] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.303913 1168 raft_consensus.cc:2241] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:21.303911 985 leader_election.cc:343] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 829221b8a79f460d90dd24be6a2d4c43
I20250625 13:52:21.304016 2638 raft_consensus.cc:2802] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.304064 1168 raft_consensus.cc:2270] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.304137 2638 raft_consensus.cc:695] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:21.304383 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829221b8a79f460d90dd24be6a2d4c43" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.304365 2638 consensus_queue.cc:237] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.304533 915 raft_consensus.cc:3058] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.304558 1168 ts_tablet_manager.cc:1905] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.307041 915 raft_consensus.cc:2466] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.309487 2254 ts_tablet_manager.cc:1939] T 50786a4c25274ffb8f9910c5dafe3933 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.310150 983 leader_election.cc:304] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.310266 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1422) took 1360 ms. Trace:
I20250625 13:52:21.310371 2254 rpcz_store.cc:276] 0625 13:52:19.949653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.949711 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:21.243117 (+1293406us) ts_tablet_manager.cc:1145] Deleting tablet 50786a4c25274ffb8f9910c5dafe3933
0625 13:52:21.243126 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.253823 (+ 10697us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.299034 (+ 45211us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.310211 (+ 11177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293214}
I20250625 13:52:21.310817 2636 raft_consensus.cc:2802] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.310894 1171 tablet_bootstrap.cc:492] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.310972 2636 raft_consensus.cc:695] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:21.311204 2636 consensus_queue.cc:237] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.312233 2594 raft_consensus.cc:491] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.312368 2594 raft_consensus.cc:513] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.312716 1168 ts_tablet_manager.cc:1918] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.312815 1168 log.cc:1199] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/8e2bc4df39674dc3be0cd32c68f1ebca
I20250625 13:52:21.312997 2594 leader_election.cc:290] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.313097 1168 ts_tablet_manager.cc:1939] T 8e2bc4df39674dc3be0cd32c68f1ebca P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.313863 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.314359 711 leader_election.cc:343] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.314958 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.315922 711 leader_election.cc:343] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.316025 711 leader_election.cc:304] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.316380 2594 raft_consensus.cc:2747] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.317070 2254 tablet_replica.cc:331] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.317339 2594 raft_consensus.cc:491] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.317487 2254 raft_consensus.cc:2241] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:21.317461 2594 raft_consensus.cc:513] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.317790 2254 pending_rounds.cc:70] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:21.317986 2594 leader_election.cc:290] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.318508 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.318898 711 leader_election.cc:343] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.317265 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1302) took 3975 ms. Trace:
I20250625 13:52:21.319136 1168 rpcz_store.cc:276] 0625 13:52:17.342232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.343084 (+ 852us) service_pool.cc:225] Handling call
0625 13:52:21.303429 (+3960345us) ts_tablet_manager.cc:1145] Deleting tablet 8e2bc4df39674dc3be0cd32c68f1ebca
0625 13:52:21.303675 (+ 246us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.308870 (+ 5195us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.312626 (+ 3756us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.317248 (+ 4622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3960199}
I20250625 13:52:21.319612 1168 tablet_replica.cc:331] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.319686 1171 tablet_bootstrap.cc:654] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.319823 1168 raft_consensus.cc:2241] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.319945 1168 raft_consensus.cc:2270] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.320391 1168 ts_tablet_manager.cc:1905] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.318908 2254 pending_rounds.cc:77] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520679235936256 op_type: NO_OP noop_request { }
I20250625 13:52:21.320816 2254 raft_consensus.cc:2887] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:21.321010 2254 raft_consensus.cc:2270] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.322010 2254 ts_tablet_manager.cc:1905] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.318498 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.324810 711 leader_election.cc:343] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.324914 711 leader_election.cc:304] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.325326 2594 raft_consensus.cc:2747] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.326320 1168 ts_tablet_manager.cc:1918] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.332798 2594 raft_consensus.cc:491] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.333034 2594 raft_consensus.cc:513] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.333824 2594 leader_election.cc:290] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.334762 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5780d88dea41e7b855b41f12e96c52" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.335287 711 leader_election.cc:343] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.335990 1170 tablet_bootstrap.cc:492] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.336202 1170 ts_tablet_manager.cc:1397] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.003s
I20250625 13:52:21.337188 1170 raft_consensus.cc:357] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.337488 1170 raft_consensus.cc:383] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.337630 1170 raft_consensus.cc:738] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.337977 1170 consensus_queue.cc:260] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.343842 1170 ts_tablet_manager.cc:1428] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 13:52:21.348004 1170 tablet_bootstrap.cc:492] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.348642 1166 tablet_bootstrap.cc:492] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.335621 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5780d88dea41e7b855b41f12e96c52" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.350235 711 leader_election.cc:343] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.350353 711 leader_election.cc:304] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.350759 2594 raft_consensus.cc:2747] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.351008 1168 log.cc:1199] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/f3c75deb64c74f0282efeed827659ec2
I20250625 13:52:21.351369 1166 tablet_bootstrap.cc:654] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.351442 1168 ts_tablet_manager.cc:1939] T f3c75deb64c74f0282efeed827659ec2 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.352288 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1319) took 3730 ms. Trace:
I20250625 13:52:21.353371 1170 tablet_bootstrap.cc:654] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.359190 1044 raft_consensus.cc:3058] T 38770320fc6541c39e10448978f8a3b0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.361855 2254 ts_tablet_manager.cc:1918] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:21.362391 2254 log.cc:1199] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/f3c75deb64c74f0282efeed827659ec2
I20250625 13:52:21.363969 1168 rpcz_store.cc:276] 0625 13:52:17.621593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.621666 (+ 73us) service_pool.cc:225] Handling call
0625 13:52:21.319572 (+3697906us) ts_tablet_manager.cc:1145] Deleting tablet f3c75deb64c74f0282efeed827659ec2
0625 13:52:21.319581 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.323430 (+ 3849us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.326238 (+ 2808us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.352265 (+ 26027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3697709}
I20250625 13:52:21.364511 1168 tablet_replica.cc:331] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.364730 1168 raft_consensus.cc:2241] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.364928 1168 raft_consensus.cc:2270] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.365769 1168 ts_tablet_manager.cc:1905] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.371748 2254 ts_tablet_manager.cc:1939] T f3c75deb64c74f0282efeed827659ec2 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.380728 1170 tablet_bootstrap.cc:492] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.380970 1170 ts_tablet_manager.cc:1397] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250625 13:52:21.381721 1170 raft_consensus.cc:357] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.381980 1170 raft_consensus.cc:383] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.382083 1170 raft_consensus.cc:738] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.382359 1170 consensus_queue.cc:260] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.384588 2580 raft_consensus.cc:491] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.384729 2580 raft_consensus.cc:513] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.385375 2580 leader_election.cc:290] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.386003 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "553af7c7326240838f37e3861d4e3357" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.386602 849 leader_election.cc:343] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.387494 1170 ts_tablet_manager.cc:1428] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20250625 13:52:21.387645 851 leader_election.cc:343] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 553af7c7326240838f37e3861d4e3357
I20250625 13:52:21.387811 851 leader_election.cc:304] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.388280 2580 raft_consensus.cc:2747] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.392361 1166 tablet_bootstrap.cc:492] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.392586 1166 ts_tablet_manager.cc:1397] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.003s
I20250625 13:52:21.393385 1166 raft_consensus.cc:357] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.393636 1166 raft_consensus.cc:383] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.393736 1166 raft_consensus.cc:738] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.393743 1170 tablet_bootstrap.cc:492] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.393999 1166 consensus_queue.cc:260] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.387629 1168 ts_tablet_manager.cc:1918] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.395495 1168 log.cc:1199] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/32a57268a18f4f00a39f8360f9c90737
I20250625 13:52:21.395751 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1423) took 1367 ms. Trace:
I20250625 13:52:21.395952 1168 ts_tablet_manager.cc:1939] T 32a57268a18f4f00a39f8360f9c90737 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.395896 2254 rpcz_store.cc:276] 0625 13:52:20.028305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.040594 (+ 12289us) service_pool.cc:225] Handling call
0625 13:52:21.316999 (+1276405us) ts_tablet_manager.cc:1145] Deleting tablet f3c75deb64c74f0282efeed827659ec2
0625 13:52:21.317008 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.328640 (+ 11632us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.361719 (+ 33079us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.395706 (+ 33987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1276218}
I20250625 13:52:21.396530 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1424) took 1363 ms. Trace:
I20250625 13:52:21.396692 2254 rpcz_store.cc:276] 0625 13:52:20.033340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.088773 (+ 55433us) service_pool.cc:225] Handling call
0625 13:52:21.396393 (+1307620us) ts_tablet_manager.cc:1145] Deleting tablet aa555aa6a0124e5bac31c2bb83bb26ae
0625 13:52:21.396403 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.396494 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1307218}
I20250625 13:52:21.397212 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1426) took 1306 ms. Trace:
I20250625 13:52:21.396740 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1317) took 3775 ms. Trace:
I20250625 13:52:21.399114 1168 rpcz_store.cc:276] 0625 13:52:17.621412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.621696 (+ 284us) service_pool.cc:225] Handling call
0625 13:52:21.364463 (+3742767us) ts_tablet_manager.cc:1145] Deleting tablet 32a57268a18f4f00a39f8360f9c90737
0625 13:52:21.364473 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.369077 (+ 4604us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.387518 (+ 18441us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.396722 (+ 9204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3742543}
I20250625 13:52:21.399436 2254 rpcz_store.cc:276] 0625 13:52:20.091087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.091139 (+ 52us) service_pool.cc:225] Handling call
0625 13:52:21.397098 (+1305959us) ts_tablet_manager.cc:1145] Deleting tablet d12f07f4145240958d7decf21c03b6bc
0625 13:52:21.397106 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.397179 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1305691}
I20250625 13:52:21.399505 1168 tablet_replica.cc:331] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.399600 1166 ts_tablet_manager.cc:1428] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 13:52:21.399747 1168 raft_consensus.cc:2241] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.399960 1168 raft_consensus.cc:2270] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.400048 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1427) took 1308 ms. Trace:
I20250625 13:52:21.400179 2254 rpcz_store.cc:276] 0625 13:52:20.091185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.091360 (+ 175us) service_pool.cc:225] Handling call
0625 13:52:21.399928 (+1308568us) ts_tablet_manager.cc:1145] Deleting tablet 5af2b974df3a41f7b60537c9d8eac1e6
0625 13:52:21.399936 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.400018 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308369}
I20250625 13:52:21.399703 1170 tablet_bootstrap.cc:654] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.400617 1168 ts_tablet_manager.cc:1905] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.401654 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1425) took 1310 ms. Trace:
I20250625 13:52:21.401784 2254 rpcz_store.cc:276] 0625 13:52:20.090975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.091591 (+ 616us) service_pool.cc:225] Handling call
0625 13:52:21.401534 (+1309943us) ts_tablet_manager.cc:1145] Deleting tablet 78c79a2f6544493ba3f3ec20ea67fa95
0625 13:52:21.401552 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.401620 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309748}
I20250625 13:52:21.402245 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1428) took 1254 ms. Trace:
I20250625 13:52:21.402385 2254 rpcz_store.cc:276] 0625 13:52:20.148152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.148197 (+ 45us) service_pool.cc:225] Handling call
0625 13:52:21.402143 (+1253946us) ts_tablet_manager.cc:1145] Deleting tablet d406206ea8e14121bf43214adfc5afda
0625 13:52:21.402151 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.402215 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253718}
I20250625 13:52:21.402823 2254 tablet_replica.cc:331] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.406823 1168 ts_tablet_manager.cc:1918] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.407454 1166 tablet_bootstrap.cc:492] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.407503 2254 raft_consensus.cc:2241] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:21.409174 1166 tablet_bootstrap.cc:654] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.409464 2254 pending_rounds.cc:70] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:21.409583 2254 pending_rounds.cc:77] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520680225406976 op_type: NO_OP noop_request { }
I20250625 13:52:21.409737 2254 raft_consensus.cc:2887] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:21.409890 2254 raft_consensus.cc:2270] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.410668 2254 ts_tablet_manager.cc:1905] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.412235 2580 raft_consensus.cc:491] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.414577 1168 log.cc:1199] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/b7d888e628a94f939ecf2571e60b1948
I20250625 13:52:21.414618 2580 raft_consensus.cc:513] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.414896 1168 ts_tablet_manager.cc:1939] T b7d888e628a94f939ecf2571e60b1948 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.415359 2580 leader_election.cc:290] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.415638 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1318) took 3794 ms. Trace:
I20250625 13:52:21.416421 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2d6f05348e43339e7167a9cddca2ba" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.416625 1044 raft_consensus.cc:2466] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:21.418242 849 leader_election.cc:304] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:21.418752 851 leader_election.cc:343] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4d2d6f05348e43339e7167a9cddca2ba
I20250625 13:52:21.419414 2580 raft_consensus.cc:2802] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.419499 2580 raft_consensus.cc:491] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.419575 2580 raft_consensus.cc:3058] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.415712 1168 rpcz_store.cc:276] 0625 13:52:17.621514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.621962 (+ 448us) service_pool.cc:225] Handling call
0625 13:52:21.399456 (+3777494us) ts_tablet_manager.cc:1145] Deleting tablet b7d888e628a94f939ecf2571e60b1948
0625 13:52:21.399468 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.403878 (+ 4410us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.406733 (+ 2855us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.415617 (+ 8884us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3777360}
I20250625 13:52:21.419881 1168 tablet_replica.cc:331] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.420046 1168 raft_consensus.cc:2241] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.420171 1168 raft_consensus.cc:2270] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.420652 1168 ts_tablet_manager.cc:1905] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.422096 2580 raft_consensus.cc:513] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.422659 2580 leader_election.cc:290] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.423009 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2d6f05348e43339e7167a9cddca2ba" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:21.423175 1044 raft_consensus.cc:3058] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.425760 1170 tablet_bootstrap.cc:492] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.425879 1170 ts_tablet_manager.cc:1397] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250625 13:52:21.425910 1044 raft_consensus.cc:2466] T 4d2d6f05348e43339e7167a9cddca2ba P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:21.426534 849 leader_election.cc:304] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.426539 1170 raft_consensus.cc:357] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.426700 1170 raft_consensus.cc:383] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.426756 1170 raft_consensus.cc:738] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.426786 2580 raft_consensus.cc:2802] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.426956 1170 consensus_queue.cc:260] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.427397 1170 ts_tablet_manager.cc:1428] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 13:52:21.427582 2580 raft_consensus.cc:695] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:21.427663 1170 tablet_bootstrap.cc:492] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:21.427861 851 leader_election.cc:343] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4d2d6f05348e43339e7167a9cddca2ba
I20250625 13:52:21.427927 2580 consensus_queue.cc:237] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.429169 1170 tablet_bootstrap.cc:654] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.431594 1171 tablet_bootstrap.cc:492] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.431718 1171 ts_tablet_manager.cc:1397] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.121s user 0.024s sys 0.020s
I20250625 13:52:21.432353 1171 raft_consensus.cc:357] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.432504 1171 raft_consensus.cc:383] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.432552 1171 raft_consensus.cc:738] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.432716 1171 consensus_queue.cc:260] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.433151 1171 ts_tablet_manager.cc:1428] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 13:52:21.433396 1171 tablet_bootstrap.cc:492] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.434976 1171 tablet_bootstrap.cc:654] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.443048 1166 tablet_bootstrap.cc:492] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.452035 1166 ts_tablet_manager.cc:1397] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20250625 13:52:21.452155 1170 tablet_bootstrap.cc:492] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.452299 1170 ts_tablet_manager.cc:1397] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250625 13:52:21.454309 1166 raft_consensus.cc:357] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.454506 1168 ts_tablet_manager.cc:1918] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.460392 1170 raft_consensus.cc:357] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.460614 1170 raft_consensus.cc:383] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.460669 1170 raft_consensus.cc:738] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.460877 1170 consensus_queue.cc:260] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.461340 1170 ts_tablet_manager.cc:1428] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 13:52:21.461618 1170 tablet_bootstrap.cc:492] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.463191 1170 tablet_bootstrap.cc:654] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.463519 1166 raft_consensus.cc:383] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.464279 1168 log.cc:1199] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d26820eaba98401da3a8b5348be21fc6
I20250625 13:52:21.464625 1168 ts_tablet_manager.cc:1939] T d26820eaba98401da3a8b5348be21fc6 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.466018 2636 raft_consensus.cc:491] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.466131 2636 raft_consensus.cc:513] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.466389 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1321) took 3800 ms. Trace:
I20250625 13:52:21.466472 1168 rpcz_store.cc:276] 0625 13:52:17.665878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.665934 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:21.419837 (+3753903us) ts_tablet_manager.cc:1145] Deleting tablet d26820eaba98401da3a8b5348be21fc6
0625 13:52:21.419846 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.445417 (+ 25571us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.454394 (+ 8977us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.466370 (+ 11976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3753702}
I20250625 13:52:21.466681 2636 leader_election.cc:290] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.467321 1171 tablet_bootstrap.cc:492] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
W20250625 13:52:21.467389 985 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:52:21.467510 1171 ts_tablet_manager.cc:1397] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
W20250625 13:52:21.468149 983 leader_election.cc:343] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:52:21.468271 983 leader_election.cc:304] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.468274 1171 raft_consensus.cc:357] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.468406 1171 raft_consensus.cc:383] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.468451 1171 raft_consensus.cc:738] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.468633 1171 consensus_queue.cc:260] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.468766 2638 raft_consensus.cc:2747] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.469077 1171 ts_tablet_manager.cc:1428] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250625 13:52:21.469933 849 consensus_peers.cc:487] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50786a4c25274ffb8f9910c5dafe3933. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:21.470124 2638 raft_consensus.cc:491] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.470247 2638 raft_consensus.cc:513] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.470707 2638 leader_election.cc:290] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:21.471941 985 leader_election.cc:343] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 2de9656958164dc4ae2fb0078b38f6c3
I20250625 13:52:21.472462 1171 tablet_bootstrap.cc:492] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.473392 1168 tablet_replica.cc:331] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.473590 1168 raft_consensus.cc:2241] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.473734 1168 raft_consensus.cc:2270] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.474220 1168 ts_tablet_manager.cc:1905] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.474485 1171 tablet_bootstrap.cc:654] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:21.475756 983 leader_election.cc:343] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 2de9656958164dc4ae2fb0078b38f6c3
I20250625 13:52:21.475852 983 leader_election.cc:304] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.463600 1166 raft_consensus.cc:738] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.476136 1166 consensus_queue.cc:260] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.479645 2638 raft_consensus.cc:2747] T 2de9656958164dc4ae2fb0078b38f6c3 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.489233 1166 ts_tablet_manager.cc:1428] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20250625 13:52:21.490280 2638 raft_consensus.cc:491] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.490448 2638 raft_consensus.cc:513] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.491015 2638 leader_election.cc:290] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:21.492170 985 leader_election.cc:343] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: dfaba56442c0498da95e6fad126ab86d
W20250625 13:52:21.493206 983 leader_election.cc:343] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: dfaba56442c0498da95e6fad126ab86d
I20250625 13:52:21.493316 983 leader_election.cc:304] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.497902 1166 tablet_bootstrap.cc:492] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.507506 1166 tablet_bootstrap.cc:654] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.508875 2638 raft_consensus.cc:2747] T dfaba56442c0498da95e6fad126ab86d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.509251 1044 raft_consensus.cc:3058] T e2e6fbf3d81547f798b246a8b8f61387 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.511729 1171 tablet_bootstrap.cc:492] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.511866 1171 ts_tablet_manager.cc:1397] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.002s
I20250625 13:52:21.512377 1168 ts_tablet_manager.cc:1918] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.512506 1168 log.cc:1199] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/413afd7b62d24fe8b9eceadd57e29fc8
I20250625 13:52:21.512838 1168 ts_tablet_manager.cc:1939] T 413afd7b62d24fe8b9eceadd57e29fc8 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.513150 2594 consensus_queue.cc:1035] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:21.513872 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1322) took 3847 ms. Trace:
I20250625 13:52:21.513967 1168 rpcz_store.cc:276] 0625 13:52:17.666007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.666054 (+ 47us) service_pool.cc:225] Handling call
0625 13:52:21.470193 (+3804139us) ts_tablet_manager.cc:1145] Deleting tablet 413afd7b62d24fe8b9eceadd57e29fc8
0625 13:52:21.473343 (+ 3150us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.508879 (+ 35536us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.512286 (+ 3407us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.513849 (+ 1563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3804028}
I20250625 13:52:21.512728 1171 raft_consensus.cc:357] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.515408 1171 raft_consensus.cc:383] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.515499 1171 raft_consensus.cc:738] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.515705 1171 consensus_queue.cc:260] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.520148 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1324) took 3651 ms. Trace:
I20250625 13:52:21.520249 1168 rpcz_store.cc:276] 0625 13:52:17.868998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.869055 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:21.520020 (+3650965us) ts_tablet_manager.cc:1145] Deleting tablet 7d7b671a935b43a6a80a37d38e57f230
0625 13:52:21.520030 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.520129 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3650754}
I20250625 13:52:21.521289 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1323) took 3653 ms. Trace:
I20250625 13:52:21.521370 1168 rpcz_store.cc:276] 0625 13:52:17.867811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.871766 (+ 3955us) service_pool.cc:225] Handling call
0625 13:52:21.521194 (+3649428us) ts_tablet_manager.cc:1145] Deleting tablet 14087755cd5744958499f32fd77b655b
0625 13:52:21.521203 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.521274 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3649212}
I20250625 13:52:21.522002 1171 ts_tablet_manager.cc:1428] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250625 13:52:21.522171 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1325) took 3637 ms. Trace:
I20250625 13:52:21.522238 1168 rpcz_store.cc:276] 0625 13:52:17.884666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.884890 (+ 224us) service_pool.cc:225] Handling call
0625 13:52:21.522075 (+3637185us) ts_tablet_manager.cc:1145] Deleting tablet 6cabec757d6749df9a8aae136342febf
0625 13:52:21.522088 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.522157 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":3637016}
I20250625 13:52:21.522562 2631 raft_consensus.cc:491] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.522667 2631 raft_consensus.cc:513] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.523222 2631 leader_election.cc:290] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.523424 1171 tablet_bootstrap.cc:492] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.523698 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d51b886302149839efc6f547b944639" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.523810 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d51b886302149839efc6f547b944639" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:21.523993 764 raft_consensus.cc:2466] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:21.524271 849 leader_election.cc:343] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.524413 851 leader_election.cc:304] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:21.524824 2631 raft_consensus.cc:491] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.524832 2580 raft_consensus.cc:2802] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.524924 2580 raft_consensus.cc:491] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.524973 2580 raft_consensus.cc:3058] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.524947 2631 raft_consensus.cc:513] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.525050 1171 tablet_bootstrap.cc:654] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.525478 2631 leader_election.cc:290] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.525976 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71cf3b95139245a4ba11b6bcaa3ab3f4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:21.526144 764 raft_consensus.cc:2466] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:21.526562 851 leader_election.cc:304] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.526829 2631 raft_consensus.cc:2802] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.526901 2631 raft_consensus.cc:491] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.526970 2631 raft_consensus.cc:3058] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.527956 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71cf3b95139245a4ba11b6bcaa3ab3f4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.528126 1045 raft_consensus.cc:2466] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:21.529145 2580 raft_consensus.cc:513] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.529547 2631 raft_consensus.cc:513] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.529752 2580 leader_election.cc:290] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.530014 2631 leader_election.cc:290] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.530761 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71cf3b95139245a4ba11b6bcaa3ab3f4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:21.530944 1048 raft_consensus.cc:3058] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.533687 1048 raft_consensus.cc:2466] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:21.534006 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d51b886302149839efc6f547b944639" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:21.534308 2638 raft_consensus.cc:491] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.534428 2638 raft_consensus.cc:513] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.535010 2638 leader_election.cc:290] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.535071 1166 tablet_bootstrap.cc:492] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.535207 1166 ts_tablet_manager.cc:1397] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250625 13:52:21.535988 1166 raft_consensus.cc:357] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:21.536063 985 leader_election.cc:343] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 569f6cba69394f2fa005686056f19929
I20250625 13:52:21.536147 1166 raft_consensus.cc:383] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.536213 1166 raft_consensus.cc:738] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.536407 1166 consensus_queue.cc:260] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.536914 1166 ts_tablet_manager.cc:1428] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:21.537520 763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71cf3b95139245a4ba11b6bcaa3ab3f4" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:21.537659 763 raft_consensus.cc:3058] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.537767 1171 tablet_bootstrap.cc:492] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.537914 1171 ts_tablet_manager.cc:1397] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250625 13:52:21.538019 2254 ts_tablet_manager.cc:1918] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:21.538115 2254 log.cc:1199] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d9462da7574a46839763b1c388102e60
I20250625 13:52:21.538421 2254 ts_tablet_manager.cc:1939] T d9462da7574a46839763b1c388102e60 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.538667 1171 raft_consensus.cc:357] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.538808 1171 raft_consensus.cc:383] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.538856 1171 raft_consensus.cc:738] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.539063 1171 consensus_queue.cc:260] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.539198 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1429) took 1300 ms. Trace:
I20250625 13:52:21.539667 1171 ts_tablet_manager.cc:1428] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:21.540109 1170 tablet_bootstrap.cc:492] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.540138 763 raft_consensus.cc:2466] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:21.540232 1170 ts_tablet_manager.cc:1397] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.004s
I20250625 13:52:21.540421 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d51b886302149839efc6f547b944639" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:21.540608 764 raft_consensus.cc:3058] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.541072 1170 raft_consensus.cc:357] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.541244 1170 raft_consensus.cc:383] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.541298 1170 raft_consensus.cc:738] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.541548 1166 tablet_bootstrap.cc:492] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.541548 1170 consensus_queue.cc:260] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.542302 1170 ts_tablet_manager.cc:1428] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:21.542582 1170 tablet_bootstrap.cc:492] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.544248 1170 tablet_bootstrap.cc:654] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.545020 849 leader_election.cc:304] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:21.545287 849 leader_election.cc:343] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:21.545680 983 leader_election.cc:343] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 569f6cba69394f2fa005686056f19929
I20250625 13:52:21.545810 983 leader_election.cc:304] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.546187 2638 raft_consensus.cc:2747] T 569f6cba69394f2fa005686056f19929 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.549660 1166 tablet_bootstrap.cc:654] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.550931 2631 raft_consensus.cc:2802] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.551067 2631 raft_consensus.cc:695] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:21.551296 2631 consensus_queue.cc:237] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.543277 764 raft_consensus.cc:2466] T 7d51b886302149839efc6f547b944639 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:21.539270 2254 rpcz_store.cc:276] 0625 13:52:20.238659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.239230 (+ 571us) service_pool.cc:225] Handling call
0625 13:52:21.402757 (+1163527us) ts_tablet_manager.cc:1145] Deleting tablet d9462da7574a46839763b1c388102e60
0625 13:52:21.402765 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.518132 (+115367us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.537913 (+ 19781us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.539180 (+ 1267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163317}
I20250625 13:52:21.555876 851 leader_election.cc:304] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:21.556219 2573 raft_consensus.cc:2802] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.556195 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1431) took 1206 ms. Trace:
I20250625 13:52:21.556310 2573 raft_consensus.cc:695] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:21.556288 2254 rpcz_store.cc:276] 0625 13:52:20.350165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.350323 (+ 158us) service_pool.cc:225] Handling call
0625 13:52:21.556062 (+1205739us) ts_tablet_manager.cc:1145] Deleting tablet e29bade90b184d2483cb1fe85033a03c
0625 13:52:21.556073 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.556176 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205568}
I20250625 13:52:21.556586 2254 tablet_replica.cc:331] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.556557 2573 consensus_queue.cc:237] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.556813 2254 raft_consensus.cc:2241] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.556991 2254 raft_consensus.cc:2270] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.557551 2254 ts_tablet_manager.cc:1905] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.563055 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1326) took 3586 ms. Trace:
I20250625 13:52:21.563200 1171 tablet_bootstrap.cc:492] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.564941 1171 tablet_bootstrap.cc:654] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.563212 1168 rpcz_store.cc:276] 0625 13:52:17.976172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.976229 (+ 57us) service_pool.cc:225] Handling call
0625 13:52:21.531071 (+3554842us) ts_tablet_manager.cc:1145] Deleting tablet 8e4df287c95946e2beb94524073e5ee2
0625 13:52:21.562898 (+ 31827us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.563035 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3534025}
I20250625 13:52:21.566412 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1327) took 3587 ms. Trace:
I20250625 13:52:21.566558 1168 rpcz_store.cc:276] 0625 13:52:17.978489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:17.978678 (+ 189us) service_pool.cc:225] Handling call
0625 13:52:21.566314 (+3587636us) ts_tablet_manager.cc:1145] Deleting tablet 663588b7e9ee430faa180fec00897e22
0625 13:52:21.566322 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.566398 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3569217}
I20250625 13:52:21.567047 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1328) took 3296 ms. Trace:
I20250625 13:52:21.567154 1168 rpcz_store.cc:276] 0625 13:52:18.270419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.270487 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:21.566965 (+3296478us) ts_tablet_manager.cc:1145] Deleting tablet d92fe17f576e491eb189a10ca72a7e09
0625 13:52:21.566972 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.567033 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258045}
I20250625 13:52:21.567273 1170 tablet_bootstrap.cc:492] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.567445 1170 ts_tablet_manager.cc:1397] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20250625 13:52:21.568193 1170 raft_consensus.cc:357] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.568358 1170 raft_consensus.cc:383] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.568420 1170 raft_consensus.cc:738] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.568646 1170 consensus_queue.cc:260] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.569432 1170 ts_tablet_manager.cc:1428] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.570204 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1329) took 3283 ms. Trace:
I20250625 13:52:21.570319 1168 rpcz_store.cc:276] 0625 13:52:18.286688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.286775 (+ 87us) service_pool.cc:225] Handling call
0625 13:52:21.570094 (+3283319us) ts_tablet_manager.cc:1145] Deleting tablet 6c89a899837f40a8ac9eed9f1d68e420
0625 13:52:21.570103 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.570182 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3260089}
I20250625 13:52:21.570873 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1331) took 3281 ms. Trace:
I20250625 13:52:21.571132 1168 rpcz_store.cc:276] 0625 13:52:18.289038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.289100 (+ 62us) service_pool.cc:225] Handling call
0625 13:52:21.570782 (+3281682us) ts_tablet_manager.cc:1145] Deleting tablet 8dd0d216fb394f31afa4478ad2097fe5
0625 13:52:21.570790 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.570859 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3260740}
I20250625 13:52:21.571893 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1330) took 3283 ms. Trace:
I20250625 13:52:21.572865 2580 raft_consensus.cc:491] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.572954 1168 rpcz_store.cc:276] 0625 13:52:18.288064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.288118 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:21.571799 (+3283681us) ts_tablet_manager.cc:1145] Deleting tablet 8a9b6e6e1f2d41288a958115dc18d4fb
0625 13:52:21.571808 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.571879 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":377,"tablet-delete.queue_time_us":3261770}
I20250625 13:52:21.573071 2580 raft_consensus.cc:513] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.573767 2580 leader_election.cc:290] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.574930 2254 ts_tablet_manager.cc:1918] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:21.575073 2254 log.cc:1199] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/4c49ef682cbb46e1ae5c3b91e4e2d9c9
I20250625 13:52:21.575518 2254 ts_tablet_manager.cc:1939] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.582667 2638 raft_consensus.cc:491] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.582834 2638 raft_consensus.cc:513] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.583499 1171 tablet_bootstrap.cc:492] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.583549 2638 leader_election.cc:290] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.583654 1171 ts_tablet_manager.cc:1397] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.002s
I20250625 13:52:21.584470 1171 raft_consensus.cc:357] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:21.585069 985 leader_election.cc:343] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 0e3ef7f5b474466bbc19fd31c6b6d99c
I20250625 13:52:21.586766 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2c3757c0284aaca70ec317c05195d6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.586946 1046 raft_consensus.cc:2466] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:21.575012 851 leader_election.cc:343] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ef2c3757c0284aaca70ec317c05195d6
I20250625 13:52:21.574041 1168 tablet_replica.cc:331] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.587949 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e3ef7f5b474466bbc19fd31c6b6d99c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.588218 1168 raft_consensus.cc:2241] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.588253 915 raft_consensus.cc:2466] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:21.588362 1168 raft_consensus.cc:2270] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.588891 1048 raft_consensus.cc:3058] T ed3c177bb6dd488aa54bee2a447016ff P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.588922 1171 raft_consensus.cc:383] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.589076 1171 raft_consensus.cc:738] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.592092 983 leader_election.cc:304] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.592450 2638 raft_consensus.cc:2802] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.592531 2638 raft_consensus.cc:491] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.592590 2638 raft_consensus.cc:3058] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.594348 1171 consensus_queue.cc:260] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.594810 1171 ts_tablet_manager.cc:1428] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 13:52:21.589293 1168 ts_tablet_manager.cc:1905] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.594980 2638 raft_consensus.cc:513] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.596662 2638 leader_election.cc:290] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:21.596788 985 leader_election.cc:343] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 0e3ef7f5b474466bbc19fd31c6b6d99c
I20250625 13:52:21.597898 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e3ef7f5b474466bbc19fd31c6b6d99c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.598057 915 raft_consensus.cc:3058] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.600812 915 raft_consensus.cc:2466] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.601166 983 leader_election.cc:304] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.602023 2638 raft_consensus.cc:2802] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.602144 2638 raft_consensus.cc:695] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:21.602363 2638 consensus_queue.cc:237] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.604387 2636 raft_consensus.cc:491] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.604501 2636 raft_consensus.cc:513] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.604988 2636 leader_election.cc:290] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.605666 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eafc402d809245d189f3977162506ef8" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.605759 1168 ts_tablet_manager.cc:1918] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.605845 1168 log.cc:1199] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/3249c5c2aa564027a2c594ca4c31e1c8
I20250625 13:52:21.605867 915 raft_consensus.cc:2466] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:21.606156 1168 ts_tablet_manager.cc:1939] T 3249c5c2aa564027a2c594ca4c31e1c8 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.606235 983 leader_election.cc:304] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:21.606570 985 leader_election.cc:343] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: eafc402d809245d189f3977162506ef8
I20250625 13:52:21.606983 2636 raft_consensus.cc:2802] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.607067 2636 raft_consensus.cc:491] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.607139 2636 raft_consensus.cc:3058] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.610224 2636 raft_consensus.cc:513] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:21.611454 985 leader_election.cc:343] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: eafc402d809245d189f3977162506ef8
I20250625 13:52:21.611809 2636 leader_election.cc:290] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.612923 1170 tablet_bootstrap.cc:492] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.615619 1170 tablet_bootstrap.cc:654] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.613124 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1430) took 1263 ms. Trace:
I20250625 13:52:21.606817 2631 raft_consensus.cc:491] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.615816 2254 rpcz_store.cc:276] 0625 13:52:20.350029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.350523 (+ 494us) service_pool.cc:225] Handling call
0625 13:52:21.556544 (+1206021us) ts_tablet_manager.cc:1145] Deleting tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9
0625 13:52:21.556554 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.571346 (+ 14792us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.574777 (+ 3431us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.613109 (+ 38332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205877}
I20250625 13:52:21.616086 2631 raft_consensus.cc:513] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.616291 2254 tablet_replica.cc:331] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.616458 2254 raft_consensus.cc:2241] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:21.616683 2254 pending_rounds.cc:70] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Trying to abort 1 pending ops.
I20250625 13:52:21.616679 2631 leader_election.cc:290] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.616708 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1332) took 3267 ms. Trace:
I20250625 13:52:21.616725 2254 pending_rounds.cc:77] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171520679060377600 op_type: NO_OP noop_request { }
I20250625 13:52:21.616801 1168 rpcz_store.cc:276] 0625 13:52:18.348950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.349003 (+ 53us) service_pool.cc:225] Handling call
0625 13:52:21.573612 (+3224609us) ts_tablet_manager.cc:1145] Deleting tablet 3249c5c2aa564027a2c594ca4c31e1c8
0625 13:52:21.574009 (+ 397us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.598567 (+ 24558us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.605665 (+ 7098us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.616658 (+ 10993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3224415}
I20250625 13:52:21.616808 2254 raft_consensus.cc:2887] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 13:52:21.617395 1168 tablet_replica.cc:331] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.617410 2254 raft_consensus.cc:2270] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.617600 1168 raft_consensus.cc:2241] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.617755 1168 raft_consensus.cc:2270] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.618235 1168 ts_tablet_manager.cc:1905] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.618521 2254 ts_tablet_manager.cc:1905] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.618813 763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd036ec08f774f5393067ce20bb07610" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:21.619067 763 raft_consensus.cc:2466] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:21.619520 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu pid 563: Timed out: CheckLeaks RPC to 127.0.104.60:43307 timed out after 1.000s (SENT)
I20250625 13:52:21.619715 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu with pid 563
I20250625 13:52:21.621452 851 leader_election.cc:304] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.621815 2631 raft_consensus.cc:2802] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.621946 2631 raft_consensus.cc:491] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.622079 2631 raft_consensus.cc:3058] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.606778 849 leader_election.cc:304] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.613353 1171 tablet_bootstrap.cc:492] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.627606 1171 tablet_bootstrap.cc:654] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.613441 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eafc402d809245d189f3977162506ef8" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.628548 915 raft_consensus.cc:3058] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.631093 915 raft_consensus.cc:2466] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.631546 983 leader_election.cc:304] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.632748 2580 raft_consensus.cc:2802] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.632899 2580 raft_consensus.cc:491] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.633044 2580 raft_consensus.cc:3058] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.633615 1168 ts_tablet_manager.cc:1918] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.633787 1168 log.cc:1199] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/829221b8a79f460d90dd24be6a2d4c43
I20250625 13:52:21.634178 1168 ts_tablet_manager.cc:1939] T 829221b8a79f460d90dd24be6a2d4c43 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.634721 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd036ec08f774f5393067ce20bb07610" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.635044 2573 consensus_queue.cc:1035] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:21.636327 1166 tablet_bootstrap.cc:492] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.636462 1166 ts_tablet_manager.cc:1397] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.095s user 0.001s sys 0.005s
I20250625 13:52:21.637099 2693 raft_consensus.cc:491] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.637215 2693 raft_consensus.cc:513] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.637821 2693 leader_election.cc:290] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.638129 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1333) took 3288 ms. Trace:
I20250625 13:52:21.638284 1168 rpcz_store.cc:276] 0625 13:52:18.349224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.349266 (+ 42us) service_pool.cc:225] Handling call
0625 13:52:21.617334 (+3268068us) ts_tablet_manager.cc:1145] Deleting tablet 829221b8a79f460d90dd24be6a2d4c43
0625 13:52:21.617344 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.625856 (+ 8512us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.633487 (+ 7631us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.638083 (+ 4596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3267930}
I20250625 13:52:21.638813 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cab4329d58404014833ae5029f8f296d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.639593 711 leader_election.cc:343] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:21.640023 849 leader_election.cc:343] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.640324 2638 raft_consensus.cc:2802] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.640908 2631 raft_consensus.cc:513] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.641407 2638 raft_consensus.cc:695] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:21.641785 2638 consensus_queue.cc:237] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.642865 763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd036ec08f774f5393067ce20bb07610" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:21.643081 763 raft_consensus.cc:3058] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.642925 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cab4329d58404014833ae5029f8f296d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.645264 2631 leader_election.cc:290] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.645834 763 raft_consensus.cc:2466] T bd036ec08f774f5393067ce20bb07610 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
W20250625 13:52:21.645931 711 leader_election.cc:343] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.646080 711 leader_election.cc:304] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.646688 2693 raft_consensus.cc:2747] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.643090 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd036ec08f774f5393067ce20bb07610" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:21.646961 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1336) took 3012 ms. Trace:
I20250625 13:52:21.647269 1168 rpcz_store.cc:276] 0625 13:52:18.634471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.634705 (+ 234us) service_pool.cc:225] Handling call
0625 13:52:21.638641 (+3003936us) ts_tablet_manager.cc:1145] Deleting tablet 2191b836bb3044c89fee13c869d9e07d
0625 13:52:21.638650 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.646861 (+ 8211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3003774}
I20250625 13:52:21.648053 2580 raft_consensus.cc:513] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:21.647168 849 leader_election.cc:343] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.648550 851 leader_election.cc:304] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:21.648838 2631 raft_consensus.cc:2802] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.649058 2631 raft_consensus.cc:695] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:21.649341 2631 consensus_queue.cc:237] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.649801 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1337) took 3015 ms. Trace:
I20250625 13:52:21.649935 1168 rpcz_store.cc:276] 0625 13:52:18.634573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.634895 (+ 322us) service_pool.cc:225] Handling call
0625 13:52:21.649645 (+3014750us) ts_tablet_manager.cc:1145] Deleting tablet 391b412dbb9e420c90aa79473bbbe8d7
0625 13:52:21.649652 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.649745 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3014654}
I20250625 13:52:21.652503 2580 leader_election.cc:290] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.652616 2693 raft_consensus.cc:491] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.652768 2693 raft_consensus.cc:513] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.652861 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2c3757c0284aaca70ec317c05195d6" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:21.653024 1045 raft_consensus.cc:3058] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.653491 2693 leader_election.cc:290] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.656311 1045 raft_consensus.cc:2466] T ef2c3757c0284aaca70ec317c05195d6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:21.656932 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af704cdf44ef42f7b5ea10b199d0f017" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.657011 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af704cdf44ef42f7b5ea10b199d0f017" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.657485 711 leader_election.cc:343] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:21.658025 711 leader_election.cc:343] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.658129 711 leader_election.cc:304] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.658269 1166 raft_consensus.cc:357] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.658485 1166 raft_consensus.cc:383] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.658581 1166 raft_consensus.cc:738] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.658627 2693 raft_consensus.cc:2747] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.658977 1166 consensus_queue.cc:260] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.659925 1166 ts_tablet_manager.cc:1428] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250625 13:52:21.660094 2636 raft_consensus.cc:491] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.660238 2636 raft_consensus.cc:513] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.660984 2636 leader_election.cc:290] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.662417 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.662981 849 leader_election.cc:304] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:21.664037 2580 raft_consensus.cc:2802] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.661405 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1338) took 3026 ms. Trace:
I20250625 13:52:21.664371 1168 rpcz_store.cc:276] 0625 13:52:18.634666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.635011 (+ 345us) service_pool.cc:225] Handling call
0625 13:52:21.661244 (+3026233us) ts_tablet_manager.cc:1145] Deleting tablet 45079deff84a41b488ccdde2f5059431
0625 13:52:21.661257 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.661357 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3026143}
I20250625 13:52:21.664783 2580 raft_consensus.cc:695] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:21.664763 1170 tablet_bootstrap.cc:492] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.664988 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1339) took 3027 ms. Trace:
I20250625 13:52:21.665047 2636 raft_consensus.cc:491] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.665127 1170 ts_tablet_manager.cc:1397] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20250625 13:52:21.665130 2580 consensus_queue.cc:237] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.665167 2636 raft_consensus.cc:513] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.665690 2636 leader_election.cc:290] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.665931 1170 raft_consensus.cc:357] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.665966 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df72a1a461aa44fe80cff6e3870da8e7" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:21.666059 1170 raft_consensus.cc:383] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.666102 1170 raft_consensus.cc:738] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.666298 1170 consensus_queue.cc:260] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.665084 1168 rpcz_store.cc:276] 0625 13:52:18.637152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.637216 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:21.664847 (+3027631us) ts_tablet_manager.cc:1145] Deleting tablet 3fc03dbe0a19470fbd0fe550159cd232
0625 13:52:21.664856 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.664936 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3027499}
I20250625 13:52:21.666733 1170 ts_tablet_manager.cc:1428] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 13:52:21.667003 1170 tablet_bootstrap.cc:492] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.666968 985 leader_election.cc:304] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.667021 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1354) took 2864 ms. Trace:
I20250625 13:52:21.667096 1168 rpcz_store.cc:276] 0625 13:52:18.802207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.802338 (+ 131us) service_pool.cc:225] Handling call
0625 13:52:21.666899 (+2864561us) ts_tablet_manager.cc:1145] Deleting tablet aafb7eef813f48fc8295fbb30a29cd0d
0625 13:52:21.666908 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.666987 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2864354}
I20250625 13:52:21.667479 2659 raft_consensus.cc:2747] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.667563 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1355) took 2845 ms. Trace:
I20250625 13:52:21.667639 1168 rpcz_store.cc:276] 0625 13:52:18.822369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.822434 (+ 65us) service_pool.cc:225] Handling call
0625 13:52:21.667438 (+2845004us) ts_tablet_manager.cc:1145] Deleting tablet 7e4fbede3f9040709ae4bdc9bbb792ad
0625 13:52:21.667447 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.667532 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2844821}
I20250625 13:52:21.668074 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1357) took 2819 ms. Trace:
I20250625 13:52:21.668181 1168 rpcz_store.cc:276] 0625 13:52:18.848337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.848401 (+ 64us) service_pool.cc:225] Handling call
0625 13:52:21.667953 (+2819552us) ts_tablet_manager.cc:1145] Deleting tablet f0a59da2129141a698692c4b31609d1d
0625 13:52:21.667962 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.668041 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2819356}
I20250625 13:52:21.668593 1168 tablet_replica.cc:331] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.668829 1168 raft_consensus.cc:2241] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.669023 1168 raft_consensus.cc:2270] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:21.669468 985 leader_election.cc:343] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 3249c5c2aa564027a2c594ca4c31e1c8
I20250625 13:52:21.669554 1168 ts_tablet_manager.cc:1905] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.669554 1171 tablet_bootstrap.cc:492] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.669845 1171 ts_tablet_manager.cc:1397] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.003s
I20250625 13:52:21.670213 1170 tablet_bootstrap.cc:654] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.670692 1171 raft_consensus.cc:357] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.674584 1171 raft_consensus.cc:383] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.674989 2693 consensus_queue.cc:1035] T e2e6fbf3d81547f798b246a8b8f61387 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:21.689993 1170 tablet_bootstrap.cc:492] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.690152 1170 ts_tablet_manager.cc:1397] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250625 13:52:21.690940 1170 raft_consensus.cc:357] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.691118 1170 raft_consensus.cc:383] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.691171 1170 raft_consensus.cc:738] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.691386 1170 consensus_queue.cc:260] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.691890 1170 ts_tablet_manager.cc:1428] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.692626 2254 ts_tablet_manager.cc:1918] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 13:52:21.692757 2254 log.cc:1199] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/d8985f259eb54ffdab29a9f6a2c5ae16
I20250625 13:52:21.693163 2254 ts_tablet_manager.cc:1939] T d8985f259eb54ffdab29a9f6a2c5ae16 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.693697 1170 tablet_bootstrap.cc:492] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.696048 1166 tablet_bootstrap.cc:492] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:21.696949 830 tablet.cc:2378] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:52:21.697774 1166 tablet_bootstrap.cc:654] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:21.673287 851 leader_election.cc:343] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: ef2c3757c0284aaca70ec317c05195d6
I20250625 13:52:21.706511 2573 consensus_queue.cc:1035] T ed3c177bb6dd488aa54bee2a447016ff P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:52:21.706631 983 leader_election.cc:343] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 3249c5c2aa564027a2c594ca4c31e1c8
I20250625 13:52:21.706755 983 leader_election.cc:304] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.710353 1170 tablet_bootstrap.cc:654] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.712226 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1432) took 1326 ms. Trace:
I20250625 13:52:21.712404 2254 rpcz_store.cc:276] 0625 13:52:20.385838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.388393 (+ 2555us) service_pool.cc:225] Handling call
0625 13:52:21.616243 (+1227850us) ts_tablet_manager.cc:1145] Deleting tablet d8985f259eb54ffdab29a9f6a2c5ae16
0625 13:52:21.616253 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.628754 (+ 12501us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.692517 (+ 63763us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.712202 (+ 19685us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18071,"tablet-delete.queue_time_us":1227633}
I20250625 13:52:21.713104 2254 tablet_replica.cc:331] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: stopping tablet replica
I20250625 13:52:21.713395 2254 raft_consensus.cc:2241] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.713600 2254 raft_consensus.cc:2270] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.714219 2254 ts_tablet_manager.cc:1905] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.715587 2594 consensus_queue.cc:1035] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:21.718614 1168 ts_tablet_manager.cc:1918] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.724043 1170 tablet_bootstrap.cc:492] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.724232 1170 ts_tablet_manager.cc:1397] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250625 13:52:21.725093 1170 raft_consensus.cc:357] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.725317 1170 raft_consensus.cc:383] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.725404 1170 raft_consensus.cc:738] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.725679 1170 consensus_queue.cc:260] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.726106 2636 raft_consensus.cc:2747] T 3249c5c2aa564027a2c594ca4c31e1c8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.726187 1170 ts_tablet_manager.cc:1428] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 13:52:21.729728 2778 log.cc:927] Time spent T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250625 13:52:21.729127 1171 raft_consensus.cc:738] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.731423 1168 log.cc:1199] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/e3939ef22ea344328721dfba67d933d1
I20250625 13:52:21.731559 1170 tablet_bootstrap.cc:492] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.731673 1171 consensus_queue.cc:260] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.732327 1171 ts_tablet_manager.cc:1428] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20250625 13:52:21.732905 1171 tablet_bootstrap.cc:492] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.733328 1170 tablet_bootstrap.cc:654] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.734701 2631 raft_consensus.cc:491] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.734910 2631 raft_consensus.cc:513] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.735702 2631 leader_election.cc:290] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.731904 1168 ts_tablet_manager.cc:1939] T e3939ef22ea344328721dfba67d933d1 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.736042 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7bd4939714a4359846345ed0521701c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.736577 849 leader_election.cc:343] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.736871 2254 ts_tablet_manager.cc:1918] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.736994 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1356) took 2901 ms. Trace:
I20250625 13:52:21.737143 1168 rpcz_store.cc:276] 0625 13:52:18.835831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.851387 (+ 15556us) service_pool.cc:225] Handling call
0625 13:52:21.668541 (+2817154us) ts_tablet_manager.cc:1145] Deleting tablet e3939ef22ea344328721dfba67d933d1
0625 13:52:21.668548 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.702582 (+ 34034us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.718466 (+ 15884us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.736955 (+ 18489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2816813}
I20250625 13:52:21.737907 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1359) took 2870 ms. Trace:
I20250625 13:52:21.738040 1168 rpcz_store.cc:276] 0625 13:52:18.867868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.867923 (+ 55us) service_pool.cc:225] Handling call
0625 13:52:21.737757 (+2869834us) ts_tablet_manager.cc:1145] Deleting tablet a89db9678dd444d5a4df85baccd62044
0625 13:52:21.737768 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.737876 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2869666}
I20250625 13:52:21.738672 1168 tablet_replica.cc:331] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.743976 2254 log.cc:1199] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-2/wal/wals/c605d5b64c164c1594b0a40cfa2c4cf6
I20250625 13:52:21.744321 1166 tablet_bootstrap.cc:492] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.744490 1166 ts_tablet_manager.cc:1397] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.001s
W20250625 13:52:21.739917 851 leader_election.cc:343] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: e7bd4939714a4359846345ed0521701c
I20250625 13:52:21.745417 851 leader_election.cc:304] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.745518 1168 raft_consensus.cc:2241] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:21.745859 1168 raft_consensus.cc:2270] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.746357 1168 ts_tablet_manager.cc:1905] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.746901 2580 raft_consensus.cc:2747] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.747287 1166 raft_consensus.cc:357] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.747764 1166 raft_consensus.cc:383] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.747875 1166 raft_consensus.cc:738] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
W20250625 13:52:21.747973 851 consensus_peers.cc:487] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50786a4c25274ffb8f9910c5dafe3933. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:21.748178 1166 consensus_queue.cc:260] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.748289 2254 ts_tablet_manager.cc:1939] T c605d5b64c164c1594b0a40cfa2c4cf6 P 1c008de91eb444f48da2c4f0b2d7a1ea: Deleting consensus metadata
I20250625 13:52:21.749104 2254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42238 (request call id 1433) took 1166 ms. Trace:
I20250625 13:52:21.747942 1170 tablet_bootstrap.cc:492] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.749094 1166 ts_tablet_manager.cc:1428] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250625 13:52:21.749387 1170 ts_tablet_manager.cc:1397] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20250625 13:52:21.749609 1166 tablet_bootstrap.cc:492] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.750126 1170 raft_consensus.cc:357] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.750290 1170 raft_consensus.cc:383] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.750380 1170 raft_consensus.cc:738] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.750613 1170 consensus_queue.cc:260] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.751133 1170 ts_tablet_manager.cc:1428] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.751456 1170 tablet_bootstrap.cc:492] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.752377 1166 tablet_bootstrap.cc:654] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.753185 1170 tablet_bootstrap.cc:654] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.757345 1171 tablet_bootstrap.cc:654] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.757232 2254 rpcz_store.cc:276] 0625 13:52:20.582244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.582412 (+ 168us) service_pool.cc:225] Handling call
0625 13:52:21.713046 (+1130634us) ts_tablet_manager.cc:1145] Deleting tablet c605d5b64c164c1594b0a40cfa2c4cf6
0625 13:52:21.713058 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.722625 (+ 9567us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.736741 (+ 14116us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.749081 (+ 12340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123169}
I20250625 13:52:21.768494 1170 tablet_bootstrap.cc:492] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.768723 1170 ts_tablet_manager.cc:1397] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20250625 13:52:21.769393 1048 raft_consensus.cc:3058] T df72a1a461aa44fe80cff6e3870da8e7 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.769516 1170 raft_consensus.cc:357] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.769698 1170 raft_consensus.cc:383] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.769769 1170 raft_consensus.cc:738] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.769974 1170 consensus_queue.cc:260] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.770170 1171 tablet_bootstrap.cc:492] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.770313 1171 ts_tablet_manager.cc:1397] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20250625 13:52:21.770550 1170 ts_tablet_manager.cc:1428] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:21.770833 1170 tablet_bootstrap.cc:492] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.772601 1170 tablet_bootstrap.cc:654] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.777902 1171 raft_consensus.cc:357] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.778152 1171 raft_consensus.cc:383] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.778223 1171 raft_consensus.cc:738] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.778463 1171 consensus_queue.cc:260] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:21.779424 1999 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.60:43307 timed out after 0.649s (SENT)
I20250625 13:52:21.788537 1171 ts_tablet_manager.cc:1428] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 13:52:21.788841 1170 tablet_bootstrap.cc:492] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.788923 1171 tablet_bootstrap.cc:492] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.789019 1170 ts_tablet_manager.cc:1397] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250625 13:52:21.789757 1170 raft_consensus.cc:357] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.790001 1170 raft_consensus.cc:383] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.790091 1170 raft_consensus.cc:738] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.790349 1170 consensus_queue.cc:260] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.790953 1170 ts_tablet_manager.cc:1428] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250625 13:52:21.791229 1170 tablet_bootstrap.cc:492] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.793099 1170 tablet_bootstrap.cc:654] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.798405 1171 tablet_bootstrap.cc:654] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.800107 2693 consensus_queue.cc:1035] T fd164b54c1d14ea98422360705cf42bf P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:21.801545 1168 ts_tablet_manager.cc:1918] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:21.801753 535 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Refusing update from remote peer 3fec9d8a79054a5991ae90d8a3101e32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1848. Preceding OpId from leader: term: 2 index: 1850. (index mismatch)
I20250625 13:52:21.802516 2744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1850, Last known committed idx: 1843, Time since last communication: 0.000s
I20250625 13:52:21.803185 2693 raft_consensus.cc:491] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.803349 2693 raft_consensus.cc:513] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.803959 2693 leader_election.cc:290] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.804193 1168 log.cc:1199] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/653267b468ef4958b98af0ed6a7bf4d3
W20250625 13:52:21.804804 1098 tablet.cc:2378] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 13:52:21.805130 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1310fee26fe642bf9919a2bd8e1c091f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.805552 711 leader_election.cc:343] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.805830 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1310fee26fe642bf9919a2bd8e1c091f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.806334 711 leader_election.cc:343] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.806422 711 leader_election.cc:304] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.806788 2693 raft_consensus.cc:2747] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.813971 1170 tablet_bootstrap.cc:492] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.814193 1170 ts_tablet_manager.cc:1397] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250625 13:52:21.815207 1168 ts_tablet_manager.cc:1939] T 653267b468ef4958b98af0ed6a7bf4d3 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:21.815179 1170 raft_consensus.cc:357] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.815445 1170 raft_consensus.cc:383] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.815562 1170 raft_consensus.cc:738] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.817284 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1358) took 2949 ms. Trace:
I20250625 13:52:21.823704 1168 rpcz_store.cc:276] 0625 13:52:18.867744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:18.868727 (+ 983us) service_pool.cc:225] Handling call
0625 13:52:21.738605 (+2869878us) ts_tablet_manager.cc:1145] Deleting tablet 653267b468ef4958b98af0ed6a7bf4d3
0625 13:52:21.738615 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.765084 (+ 26469us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.801394 (+ 36310us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.817260 (+ 15866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2869636}
W20250625 13:52:21.824644 1999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.60:43307, user_credentials={real_user=slave}} blocked reactor thread for 45283.2us
I20250625 13:52:21.825763 1170 consensus_queue.cc:260] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.827454 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1380) took 2503 ms. Trace:
I20250625 13:52:21.830133 1168 rpcz_store.cc:276] 0625 13:52:19.324411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.324467 (+ 56us) service_pool.cc:225] Handling call
0625 13:52:21.827281 (+2502814us) ts_tablet_manager.cc:1145] Deleting tablet 7e8396ea40d34c66ab740c66647a6afc
0625 13:52:21.827293 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.827434 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2502607}
I20250625 13:52:21.830632 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1381) took 2478 ms. Trace:
I20250625 13:52:21.830772 1168 rpcz_store.cc:276] 0625 13:52:19.352027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.352085 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:21.830515 (+2478430us) ts_tablet_manager.cc:1145] Deleting tablet cbd82bb415174dbdbcfafa9d6c3f3aa7
0625 13:52:21.830523 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.830613 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2478242}
I20250625 13:52:21.831353 1170 ts_tablet_manager.cc:1428] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250625 13:52:21.831689 1170 tablet_bootstrap.cc:492] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.833331 1170 tablet_bootstrap.cc:654] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.834784 1166 tablet_bootstrap.cc:492] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.833817 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1382) took 2458 ms. Trace:
I20250625 13:52:21.834933 1166 ts_tablet_manager.cc:1397] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.007s
I20250625 13:52:21.834976 1168 rpcz_store.cc:276] 0625 13:52:19.375344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.375407 (+ 63us) service_pool.cc:225] Handling call
0625 13:52:21.833716 (+2458309us) ts_tablet_manager.cc:1145] Deleting tablet d8257f79a42247b48f2736a545362f44
0625 13:52:21.833725 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.833803 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458115}
I20250625 13:52:21.835570 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1383) took 2438 ms. Trace:
I20250625 13:52:21.835935 1166 raft_consensus.cc:357] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.836117 1166 raft_consensus.cc:383] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.836180 1166 raft_consensus.cc:738] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.836364 1166 consensus_queue.cc:260] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.836994 1166 ts_tablet_manager.cc:1428] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.837247 1166 tablet_bootstrap.cc:492] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.838832 1166 tablet_bootstrap.cc:654] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:21.843690 513 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.60:43307 timed out after 0.705s (SENT) [suppressed 4 similar messages]
I20250625 13:52:21.842206 2631 raft_consensus.cc:491] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.845975 2631 raft_consensus.cc:513] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.846803 2631 leader_election.cc:290] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:21.847369 1995 connection.cc:537] server connection from 127.0.0.1:48780 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:52:21.848344 851 leader_election.cc:343] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 69c44d72516e4883aff636ca65929c11
I20250625 13:52:21.848376 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69c44d72516e4883aff636ca65929c11" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:21.835711 1168 rpcz_store.cc:276] 0625 13:52:19.397198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.397245 (+ 47us) service_pool.cc:225] Handling call
0625 13:52:21.835460 (+2438215us) ts_tablet_manager.cc:1145] Deleting tablet 7e8396ea40d34c66ab740c66647a6afc
0625 13:52:21.835470 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.835556 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2438033}
I20250625 13:52:21.850575 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fec9d8a79054a5991ae90d8a3101e32. Latest consensus state: current_term: 2 leader_uuid: "3fec9d8a79054a5991ae90d8a3101e32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:21.850880 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: This master's current role is: FOLLOWER
W20250625 13:52:21.848985 849 leader_election.cc:343] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.851322 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1389) took 2417 ms. Trace:
I20250625 13:52:21.851420 849 leader_election.cc:304] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.856173 2631 raft_consensus.cc:2747] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.857991 1171 tablet_bootstrap.cc:492] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.859233 1171 ts_tablet_manager.cc:1397] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.005s
I20250625 13:52:21.860229 1171 raft_consensus.cc:357] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.860519 1171 raft_consensus.cc:383] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.860669 1171 raft_consensus.cc:738] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.861001 1171 consensus_queue.cc:260] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.867584 1171 ts_tablet_manager.cc:1428] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 13:52:21.867838 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3fec9d8a79054a5991ae90d8a3101e32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:21.868132 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [sys.catalog]: This master's current role is: FOLLOWER
I20250625 13:52:21.851440 1168 rpcz_store.cc:276] 0625 13:52:19.433505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.433565 (+ 60us) service_pool.cc:225] Handling call
0625 13:52:21.851193 (+2417628us) ts_tablet_manager.cc:1145] Deleting tablet e1549b9e797a44f89b316a44c5e7ee2e
0625 13:52:21.851205 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.851305 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2417455}
I20250625 13:52:21.869951 1170 tablet_bootstrap.cc:492] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.870098 1170 ts_tablet_manager.cc:1397] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.004s
I20250625 13:52:21.870536 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1390) took 2435 ms. Trace:
I20250625 13:52:21.870666 1168 rpcz_store.cc:276] 0625 13:52:19.444699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.444901 (+ 202us) service_pool.cc:225] Handling call
0625 13:52:21.870400 (+2425499us) ts_tablet_manager.cc:1145] Deleting tablet e58128254acf44129f7b4e452ac03fe5
0625 13:52:21.870410 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.870518 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2425322}
I20250625 13:52:21.871088 1171 tablet_bootstrap.cc:492] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.871595 1170 raft_consensus.cc:357] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.871816 1170 raft_consensus.cc:383] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.871852 1168 tablet_replica.cc:331] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.871906 1170 raft_consensus.cc:738] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.872169 1168 raft_consensus.cc:2241] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 13:52:21.872270 986 connection.cc:537] server connection from 127.0.0.1:42238 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:21.872414 1168 raft_consensus.cc:2270] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.872525 1171 tablet_bootstrap.cc:654] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:21.872637 714 connection.cc:537] server connection from 127.0.0.1:43474 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:52:21.872982 850 connection.cc:537] server connection from 127.0.0.1:52994 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:52:21.873044 986 connection.cc:537] client connection to 127.0.104.60:43307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:52:21.873312 714 connection.cc:537] client connection to 127.0.104.60:43307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:52:21.873409 1105 connection.cc:537] client connection to 127.0.104.60:43307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:52:21.873689 513 connection.cc:537] client connection to 127.0.104.60:43307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 13:52:21.873832 513 leader_election.cc:336] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:21.873967 513 leader_election.cc:304] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58351267767f4d9a85f974ebc40db6fe; no voters: 3fec9d8a79054a5991ae90d8a3101e32, d85834f366fa42c294c9290af21ea111
I20250625 13:52:21.874058 1170 consensus_queue.cc:260] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.874341 2803 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 13:52:21.874677 1170 ts_tablet_manager.cc:1428] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20250625 13:52:21.874887 1105 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.60:43307)
I20250625 13:52:21.874969 1170 tablet_bootstrap.cc:492] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:21.875473 1105 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.60:43307)
W20250625 13:52:21.875877 1105 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.60:43307)
W20250625 13:52:21.876250 964 tablet.cc:2378] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 13:52:21.876929 1105 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.60:43307)
I20250625 13:52:21.877048 1170 tablet_bootstrap.cc:654] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:21.878422 852 connection.cc:537] client connection to 127.0.104.60:43307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:21.879101 1171 tablet_bootstrap.cc:492] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.879231 1171 ts_tablet_manager.cc:1397] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
W20250625 13:52:21.879931 512 connection.cc:537] server connection from 127.0.0.1:54568 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:21.880117 1171 raft_consensus.cc:357] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.880306 1171 raft_consensus.cc:383] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.880362 1171 raft_consensus.cc:738] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.880661 1171 consensus_queue.cc:260] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.881353 1171 ts_tablet_manager.cc:1428] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.881654 1171 tablet_bootstrap.cc:492] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.882553 1166 tablet_bootstrap.cc:492] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.882687 1166 ts_tablet_manager.cc:1397] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250625 13:52:21.883037 1171 tablet_bootstrap.cc:654] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.886788 1166 raft_consensus.cc:357] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.887099 1166 raft_consensus.cc:383] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.887212 1166 raft_consensus.cc:738] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.887542 1166 consensus_queue.cc:260] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.894678 1166 ts_tablet_manager.cc:1428] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250625 13:52:21.895181 1166 tablet_bootstrap.cc:492] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.898180 2638 raft_consensus.cc:491] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.898388 2638 raft_consensus.cc:513] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } }
I20250625 13:52:21.899467 2638 leader_election.cc:290] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:21.900575 983 leader_election.cc:343] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: e3939ef22ea344328721dfba67d933d1
W20250625 13:52:21.901603 985 leader_election.cc:343] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: e3939ef22ea344328721dfba67d933d1
I20250625 13:52:21.901752 985 leader_election.cc:304] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:21.901952 1999 connection.cc:537] client connection to 127.0.104.60:43307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:21.902374 2638 raft_consensus.cc:2747] T e3939ef22ea344328721dfba67d933d1 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.903173 2744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3fec9d8a79054a5991ae90d8a3101e32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } } }
I20250625 13:52:21.904273 1170 tablet_bootstrap.cc:492] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:21.904428 1170 ts_tablet_manager.cc:1397] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.002s
W20250625 13:52:21.902899 1999 leader_election.cc:336] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:21.905311 1170 raft_consensus.cc:357] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.905524 1170 raft_consensus.cc:383] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.905618 1170 raft_consensus.cc:738] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.905834 1170 consensus_queue.cc:260] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.908452 1166 tablet_bootstrap.cc:654] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.909452 1168 ts_tablet_manager.cc:1905] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:21.912092 1170 ts_tablet_manager.cc:1428] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 13:52:21.912400 1170 tablet_bootstrap.cc:492] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:21.912883 2638 raft_consensus.cc:491] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.913013 2638 raft_consensus.cc:513] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.913559 2744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [sys.catalog]: This master's current role is: LEADER
I20250625 13:52:21.913622 2638 leader_election.cc:290] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.914161 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ca3f3760e6e41a29d004e6afe63ec98" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.914338 915 raft_consensus.cc:2466] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:21.914793 983 leader_election.cc:304] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:21.915925 985 leader_election.cc:343] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1ca3f3760e6e41a29d004e6afe63ec98
I20250625 13:52:21.916467 2638 raft_consensus.cc:2802] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.916579 2638 raft_consensus.cc:491] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.916663 2638 raft_consensus.cc:3058] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.916841 2594 raft_consensus.cc:491] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.916985 2594 raft_consensus.cc:513] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.917542 2594 leader_election.cc:290] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.918110 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb648c45ea344fb3bac765c9e06db2b1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:21.918566 711 leader_election.cc:343] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.919004 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb648c45ea344fb3bac765c9e06db2b1" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.919428 711 leader_election.cc:343] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.919222 2638 raft_consensus.cc:513] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.919541 711 leader_election.cc:304] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.920019 2594 raft_consensus.cc:2747] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:21.920611 985 leader_election.cc:343] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 1ca3f3760e6e41a29d004e6afe63ec98
I20250625 13:52:21.921305 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ca3f3760e6e41a29d004e6afe63ec98" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.921600 915 raft_consensus.cc:3058] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.924396 915 raft_consensus.cc:2466] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.924867 983 leader_election.cc:304] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.925136 2636 raft_consensus.cc:2802] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.925243 2636 raft_consensus.cc:695] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:21.925496 2636 consensus_queue.cc:237] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:21.926654 1999 leader_election.cc:336] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 13:52:21.927579 2638 leader_election.cc:290] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.931249 1170 tablet_bootstrap.cc:654] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.934860 1166 tablet_bootstrap.cc:492] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:21.939178 1168 ts_tablet_manager.cc:1918] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:21.939406 1168 log.cc:1199] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/eafc402d809245d189f3977162506ef8
I20250625 13:52:21.939867 1168 ts_tablet_manager.cc:1939] T eafc402d809245d189f3977162506ef8 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
W20250625 13:52:21.940060 2814 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 13:52:21.874287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43307 (local address 127.0.104.3:60289)
0625 13:52:21.939736 (+ 65449us) negotiation.cc:107] Waiting for socket to connect
0625 13:52:21.939834 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":65343,"threads_started":1}
I20250625 13:52:21.941102 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1391) took 2483 ms. Trace:
I20250625 13:52:21.941284 1168 rpcz_store.cc:276] 0625 13:52:19.457458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.457544 (+ 86us) service_pool.cc:225] Handling call
0625 13:52:21.871100 (+2413556us) ts_tablet_manager.cc:1145] Deleting tablet eafc402d809245d189f3977162506ef8
0625 13:52:21.871108 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.918107 (+ 46999us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.939053 (+ 20946us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:21.941076 (+ 2023us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413332}
I20250625 13:52:21.942077 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1392) took 2484 ms. Trace:
I20250625 13:52:21.942250 1168 rpcz_store.cc:276] 0625 13:52:19.457612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.457811 (+ 199us) service_pool.cc:225] Handling call
0625 13:52:21.941938 (+2484127us) ts_tablet_manager.cc:1145] Deleting tablet fc086d416d0f475987bd67f122487353
0625 13:52:21.941952 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.942062 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2483977}
I20250625 13:52:21.942736 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1422) took 2183 ms. Trace:
I20250625 13:52:21.942867 1168 rpcz_store.cc:276] 0625 13:52:19.759578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.768633 (+ 9055us) service_pool.cc:225] Handling call
0625 13:52:21.942605 (+2173972us) ts_tablet_manager.cc:1145] Deleting tablet 3775c883f8534c6d90687152b9ca22d1
0625 13:52:21.942617 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.942720 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173778}
I20250625 13:52:21.943284 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1426) took 2104 ms. Trace:
I20250625 13:52:21.943445 1168 rpcz_store.cc:276] 0625 13:52:19.838530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.840675 (+ 2145us) service_pool.cc:225] Handling call
0625 13:52:21.943179 (+2102504us) ts_tablet_manager.cc:1145] Deleting tablet 0cf35321b38a4e81b41563f711dbb86d
0625 13:52:21.943190 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.943268 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003726}
I20250625 13:52:21.943737 2638 raft_consensus.cc:491] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.943946 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1424) took 2105 ms. Trace:
I20250625 13:52:21.943914 2638 raft_consensus.cc:513] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.944128 1168 rpcz_store.cc:276] 0625 13:52:19.838256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.840900 (+ 2644us) service_pool.cc:225] Handling call
0625 13:52:21.943777 (+2102877us) ts_tablet_manager.cc:1145] Deleting tablet 50ac647b1d704142b94da8d506319466
0625 13:52:21.943788 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.943928 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004237}
I20250625 13:52:21.945031 2638 leader_election.cc:290] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:21.945427 985 leader_election.cc:343] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 413afd7b62d24fe8b9eceadd57e29fc8
I20250625 13:52:21.944545 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1425) took 2106 ms. Trace:
W20250625 13:52:21.941819 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1391) could send its response
W20250625 13:52:21.948494 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1392) could send its response
W20250625 13:52:21.948632 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1422) could send its response
W20250625 13:52:21.948736 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1426) could send its response
W20250625 13:52:21.948769 983 leader_election.cc:343] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 413afd7b62d24fe8b9eceadd57e29fc8
W20250625 13:52:21.948850 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1424) could send its response
I20250625 13:52:21.948894 983 leader_election.cc:304] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:21.949268 2580 raft_consensus.cc:491] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.949401 2580 raft_consensus.cc:513] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.949962 2580 leader_election.cc:290] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:21.950578 2638 raft_consensus.cc:2747] T 413afd7b62d24fe8b9eceadd57e29fc8 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:21.950922 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:21.951470 849 leader_election.cc:343] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.952946 1048 raft_consensus.cc:3058] T 669de0b7c93a4efea8623dab2e462c8d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.954545 1171 tablet_bootstrap.cc:492] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:21.954654 1171 ts_tablet_manager.cc:1397] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.073s user 0.010s sys 0.001s
I20250625 13:52:21.955346 1171 raft_consensus.cc:357] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.955498 1171 raft_consensus.cc:383] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.955540 1171 raft_consensus.cc:738] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.955750 1171 consensus_queue.cc:260] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.957132 1171 ts_tablet_manager.cc:1428] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:21.958146 1171 tablet_bootstrap.cc:492] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:21.950655 763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:21.960139 763 raft_consensus.cc:2466] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:21.960515 1166 ts_tablet_manager.cc:1397] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20250625 13:52:21.962173 1166 raft_consensus.cc:357] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.962527 851 leader_election.cc:304] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:21.962865 2580 raft_consensus.cc:2802] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.962937 2580 raft_consensus.cc:491] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.963008 2580 raft_consensus.cc:3058] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.967154 1166 raft_consensus.cc:383] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:21.967232 1166 raft_consensus.cc:738] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:21.968645 1166 consensus_queue.cc:260] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:21.976322 2580 raft_consensus.cc:513] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.977373 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:21.977553 769 raft_consensus.cc:3058] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.977578 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:21.978065 849 leader_election.cc:343] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:21.978684 1166 ts_tablet_manager.cc:1428] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 13:52:21.978993 1166 tablet_bootstrap.cc:492] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:21.980532 769 raft_consensus.cc:2466] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:21.980736 1166 tablet_bootstrap.cc:654] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:21.981093 851 leader_election.cc:304] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:21.981406 2631 raft_consensus.cc:2802] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.981527 2631 raft_consensus.cc:695] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:21.981750 2631 consensus_queue.cc:237] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.945829 1168 rpcz_store.cc:276] 0625 13:52:19.838384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.840964 (+ 2580us) service_pool.cc:225] Handling call
0625 13:52:21.944438 (+2103474us) ts_tablet_manager.cc:1145] Deleting tablet ba762a8edb8847368103e5bede9e9d0b
0625 13:52:21.944447 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.944527 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004849}
I20250625 13:52:21.983150 2638 raft_consensus.cc:491] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 13:52:21.983336 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1425) could send its response
I20250625 13:52:21.983326 2638 raft_consensus.cc:513] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.983893 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1423) took 2155 ms. Trace:
I20250625 13:52:21.983956 2638 leader_election.cc:290] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.983978 1168 rpcz_store.cc:276] 0625 13:52:19.828691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.836333 (+ 7642us) service_pool.cc:225] Handling call
0625 13:52:21.983542 (+2147209us) ts_tablet_manager.cc:1145] Deleting tablet 34fe02d7b37f4d99bbae082943223e35
0625 13:52:21.983554 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:21.983872 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037131}
W20250625 13:52:21.984170 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1423) could send its response
I20250625 13:52:21.984359 1168 tablet_replica.cc:331] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:21.984711 1168 raft_consensus.cc:2241] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:21.984920 1168 raft_consensus.cc:2270] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:21.985042 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ae5f1d414794474b2178b431cbc1d44" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.985282 915 raft_consensus.cc:2466] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
W20250625 13:52:21.985589 985 leader_election.cc:343] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4ae5f1d414794474b2178b431cbc1d44
I20250625 13:52:21.985766 983 leader_election.cc:304] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.986217 2638 raft_consensus.cc:2802] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:21.986320 2638 raft_consensus.cc:491] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:21.986389 2638 raft_consensus.cc:3058] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:21.988924 2659 raft_consensus.cc:491] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:21.989060 2659 raft_consensus.cc:513] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.989121 2638 raft_consensus.cc:513] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:21.989698 2659 leader_election.cc:290] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:21.989689 2638 leader_election.cc:290] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:21.990590 985 leader_election.cc:343] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4ae5f1d414794474b2178b431cbc1d44
I20250625 13:52:21.991526 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ae5f1d414794474b2178b431cbc1d44" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:21.991689 914 raft_consensus.cc:3058] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:21.991783 985 leader_election.cc:343] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: afa1c198dd6541809843244c7cc18194
I20250625 13:52:21.991997 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afa1c198dd6541809843244c7cc18194" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:21.994216 914 raft_consensus.cc:2466] T 4ae5f1d414794474b2178b431cbc1d44 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:21.995661 983 leader_election.cc:304] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:21.998730 2636 raft_consensus.cc:2802] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:21.999711 2580 leader_election.cc:290] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.000182 2659 raft_consensus.cc:491] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.000223 2636 raft_consensus.cc:695] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.000372 2659 raft_consensus.cc:513] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.000646 2636 consensus_queue.cc:237] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.001050 2659 leader_election.cc:290] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:22.001232 983 leader_election.cc:343] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.001374 983 leader_election.cc:304] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.001760 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.001948 2659 raft_consensus.cc:2747] T afa1c198dd6541809843244c7cc18194 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.002208 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fa52e79e00c4f9c97a05c353e11ae2c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.002749 983 leader_election.cc:304] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.003294 2659 raft_consensus.cc:2747] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.003607 1170 tablet_bootstrap.cc:492] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.003757 1170 ts_tablet_manager.cc:1397] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.004s
I20250625 13:52:22.004325 1168 ts_tablet_manager.cc:1905] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.004551 1170 raft_consensus.cc:357] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.004724 1170 raft_consensus.cc:383] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.004827 1170 raft_consensus.cc:738] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.016183 1170 consensus_queue.cc:260] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.020370 1170 ts_tablet_manager.cc:1428] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 13:52:22.021610 1170 tablet_bootstrap.cc:492] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.023245 1170 tablet_bootstrap.cc:654] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.023603 2580 raft_consensus.cc:491] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.023821 2580 raft_consensus.cc:513] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.024171 1171 tablet_bootstrap.cc:654] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.025174 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5780d88dea41e7b855b41f12e96c52" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.028215 849 leader_election.cc:343] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.028728 2580 leader_election.cc:290] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.028983 2638 raft_consensus.cc:491] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.029181 2638 raft_consensus.cc:513] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.029212 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5780d88dea41e7b855b41f12e96c52" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.029513 769 raft_consensus.cc:2466] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.029954 851 leader_election.cc:304] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.030408 2638 leader_election.cc:290] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.030599 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db6e9db2375144b38ec43ffca9ca4c1e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.030781 915 raft_consensus.cc:2466] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
W20250625 13:52:22.031026 985 leader_election.cc:343] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: db6e9db2375144b38ec43ffca9ca4c1e
I20250625 13:52:22.031160 985 leader_election.cc:304] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.031414 2580 raft_consensus.cc:2802] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.031520 2580 raft_consensus.cc:491] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.031610 2580 raft_consensus.cc:3058] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.034171 2638 raft_consensus.cc:2802] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.034291 2638 raft_consensus.cc:491] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.034391 2638 raft_consensus.cc:3058] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.034217 2580 raft_consensus.cc:513] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.035449 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5780d88dea41e7b855b41f12e96c52" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:22.035899 849 leader_election.cc:343] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.037250 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5780d88dea41e7b855b41f12e96c52" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.037459 769 raft_consensus.cc:3058] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.037758 2580 leader_election.cc:290] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.040247 1171 tablet_bootstrap.cc:492] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.040589 769 raft_consensus.cc:2466] T 6f5780d88dea41e7b855b41f12e96c52 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.042557 2638 raft_consensus.cc:513] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:22.043804 985 leader_election.cc:343] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: db6e9db2375144b38ec43ffca9ca4c1e
W20250625 13:52:22.044420 1999 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3fec9d8a79054a5991ae90d8a3101e32 -> Peer d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307): Couldn't send request to peer d85834f366fa42c294c9290af21ea111. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.047192 2636 raft_consensus.cc:491] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.047350 2636 raft_consensus.cc:513] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.047973 2636 leader_election.cc:290] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.048250 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.048563 769 raft_consensus.cc:2391] T 15c81fab8e0b442e8d39556c1bce742e P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:52:22.050350 2638 leader_election.cc:290] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.043020 851 leader_election.cc:304] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.056607 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db6e9db2375144b38ec43ffca9ca4c1e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.056829 914 raft_consensus.cc:3058] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.060635 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15c81fab8e0b442e8d39556c1bce742e" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.061386 983 leader_election.cc:304] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.061682 2638 raft_consensus.cc:3058] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.062003 914 raft_consensus.cc:2466] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.062450 983 leader_election.cc:304] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.062850 2636 raft_consensus.cc:2802] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.062995 2636 raft_consensus.cc:695] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.063262 2636 consensus_queue.cc:237] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.064321 2638 raft_consensus.cc:2747] T 15c81fab8e0b442e8d39556c1bce742e P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.066164 2774 raft_consensus.cc:2802] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.066272 2774 raft_consensus.cc:695] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.066478 2774 consensus_queue.cc:237] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.069994 2775 consensus_queue.cc:1035] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.082098 1171 ts_tablet_manager.cc:1397] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.001s
I20250625 13:52:22.088974 1168 ts_tablet_manager.cc:1918] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:22.089342 1168 log.cc:1199] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/afa1c198dd6541809843244c7cc18194
I20250625 13:52:22.091620 2774 consensus_queue.cc:1035] T 15c81fab8e0b442e8d39556c1bce742e P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.104405 1171 raft_consensus.cc:357] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.104951 1171 raft_consensus.cc:383] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.105072 1171 raft_consensus.cc:738] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.105346 1171 consensus_queue.cc:260] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.105967 1171 ts_tablet_manager.cc:1428] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250625 13:52:22.107852 2573 raft_consensus.cc:491] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.117333 2573 raft_consensus.cc:513] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.117944 2573 leader_election.cc:290] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.118425 763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bc97a56aaf741648bae711cffacc6fb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.118573 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bc97a56aaf741648bae711cffacc6fb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.118644 763 raft_consensus.cc:2466] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:22.119081 849 leader_election.cc:343] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.119203 849 leader_election.cc:304] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.119403 1170 tablet_bootstrap.cc:492] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.119473 2573 raft_consensus.cc:2802] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.119550 2573 raft_consensus.cc:491] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.119583 1170 ts_tablet_manager.cc:1397] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.098s user 0.016s sys 0.005s
I20250625 13:52:22.119607 2573 raft_consensus.cc:3058] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.120471 1170 raft_consensus.cc:357] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.120699 1170 raft_consensus.cc:383] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.120782 1170 raft_consensus.cc:738] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.121055 1170 consensus_queue.cc:260] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.121640 1170 ts_tablet_manager.cc:1428] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:22.122295 2573 raft_consensus.cc:513] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.124508 2573 leader_election.cc:290] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.124951 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bc97a56aaf741648bae711cffacc6fb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.125069 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bc97a56aaf741648bae711cffacc6fb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.125219 2580 raft_consensus.cc:491] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.125241 769 raft_consensus.cc:3058] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.125360 2580 raft_consensus.cc:513] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.125677 849 leader_election.cc:343] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.126221 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9efc7631df4ad29112a41f229ae2e9" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.126559 851 leader_election.cc:343] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4c9efc7631df4ad29112a41f229ae2e9
I20250625 13:52:22.126768 2580 leader_election.cc:290] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:22.126798 849 leader_election.cc:343] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.126895 849 leader_election.cc:304] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.130044 2573 raft_consensus.cc:2747] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.131474 1170 tablet_bootstrap.cc:492] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.133314 1170 tablet_bootstrap.cc:654] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.134544 2594 consensus_queue.cc:1035] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.140233 769 raft_consensus.cc:2466] T 3bc97a56aaf741648bae711cffacc6fb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.142205 1048 raft_consensus.cc:3058] T 5a289b9e3fac4fd8bacc22c235f7010c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.142768 851 leader_election.cc:304] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.144340 2580 raft_consensus.cc:2802] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.144490 2580 raft_consensus.cc:695] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.144701 2580 consensus_queue.cc:237] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.145989 2693 consensus_queue.cc:1035] T 5a289b9e3fac4fd8bacc22c235f7010c P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.152427 1171 tablet_bootstrap.cc:492] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.154632 1171 tablet_bootstrap.cc:654] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.155375 1170 tablet_bootstrap.cc:492] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.155578 1170 ts_tablet_manager.cc:1397] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20250625 13:52:22.156355 1170 raft_consensus.cc:357] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.156570 1170 raft_consensus.cc:383] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.156672 1170 raft_consensus.cc:738] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.156911 1170 consensus_queue.cc:260] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.158321 1170 ts_tablet_manager.cc:1428] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:22.159556 1170 tablet_bootstrap.cc:492] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.089743 1168 ts_tablet_manager.cc:1939] T afa1c198dd6541809843244c7cc18194 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.161124 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1427) took 2253 ms. Trace:
I20250625 13:52:22.161324 1168 rpcz_store.cc:276] 0625 13:52:19.908136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.908217 (+ 81us) service_pool.cc:225] Handling call
0625 13:52:21.984312 (+2076095us) ts_tablet_manager.cc:1145] Deleting tablet afa1c198dd6541809843244c7cc18194
0625 13:52:21.984321 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.013464 (+ 29143us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.088830 (+ 75366us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.161099 (+ 72269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037842}
W20250625 13:52:22.161885 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1427) could send its response
I20250625 13:52:22.162223 1168 tablet_replica.cc:331] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:22.162505 1170 tablet_bootstrap.cc:654] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.162505 1168 raft_consensus.cc:2241] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:22.162784 1168 raft_consensus.cc:2270] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:22.163451 1168 ts_tablet_manager.cc:1905] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.174741 1166 tablet_bootstrap.cc:492] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.175019 1166 ts_tablet_manager.cc:1397] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.196s user 0.014s sys 0.014s
I20250625 13:52:22.176082 1166 raft_consensus.cc:357] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.176355 1166 raft_consensus.cc:383] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.176455 1170 tablet_bootstrap.cc:492] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.176478 1166 raft_consensus.cc:738] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.176662 1170 ts_tablet_manager.cc:1397] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250625 13:52:22.176909 1166 consensus_queue.cc:260] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.177841 1170 raft_consensus.cc:357] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.178027 1170 raft_consensus.cc:383] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.178097 1170 raft_consensus.cc:738] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.178334 1170 consensus_queue.cc:260] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.178851 1170 ts_tablet_manager.cc:1428] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:22.180359 1170 tablet_bootstrap.cc:492] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.185590 1166 ts_tablet_manager.cc:1428] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250625 13:52:22.185987 1166 tablet_bootstrap.cc:492] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.189898 1166 tablet_bootstrap.cc:654] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.190150 1170 tablet_bootstrap.cc:654] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.192714 2580 consensus_queue.cc:1035] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.197752 2838 raft_consensus.cc:491] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.197896 2838 raft_consensus.cc:513] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.198853 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.198913 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.199092 769 raft_consensus.cc:2466] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.199434 851 leader_election.cc:304] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:22.199609 849 leader_election.cc:343] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.199693 2774 raft_consensus.cc:2802] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.199815 2774 raft_consensus.cc:491] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.199903 2774 raft_consensus.cc:3058] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.199928 2838 leader_election.cc:290] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.200791 1171 tablet_bootstrap.cc:492] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.201016 1171 ts_tablet_manager.cc:1397] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.004s
I20250625 13:52:22.201867 1171 raft_consensus.cc:357] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.202078 1171 raft_consensus.cc:383] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.202164 1171 raft_consensus.cc:738] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.202427 1171 consensus_queue.cc:260] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.203049 1171 ts_tablet_manager.cc:1428] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:22.203410 2774 raft_consensus.cc:513] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.204707 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.204767 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.204954 769 raft_consensus.cc:3058] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:22.205413 849 leader_election.cc:343] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.206107 1171 tablet_bootstrap.cc:492] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.208200 769 raft_consensus.cc:2466] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.208606 851 leader_election.cc:304] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.208985 2573 raft_consensus.cc:491] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.209111 2573 raft_consensus.cc:513] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.209806 2573 leader_election.cc:290] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.209947 2573 raft_consensus.cc:2802] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.210062 2573 raft_consensus.cc:695] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.210341 2573 consensus_queue.cc:237] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.210577 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "189a9347cd5d4b5f955e5c0706aa3947" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.211217 849 leader_election.cc:343] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.212746 763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "189a9347cd5d4b5f955e5c0706aa3947" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.212960 763 raft_consensus.cc:2466] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.213326 2693 raft_consensus.cc:491] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.213552 2693 raft_consensus.cc:513] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.213750 851 leader_election.cc:304] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.214352 2693 leader_election.cc:290] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.214476 2838 raft_consensus.cc:2802] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.214570 2838 raft_consensus.cc:491] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.214645 2838 raft_consensus.cc:3058] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.215211 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48645c913fc24c03a922cf6ea7531022" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.215474 914 raft_consensus.cc:2466] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:22.216202 711 leader_election.cc:304] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.217538 1170 tablet_bootstrap.cc:492] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.217645 2838 raft_consensus.cc:513] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.218106 2693 raft_consensus.cc:2802] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.218221 2693 raft_consensus.cc:491] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.218319 2693 raft_consensus.cc:3058] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.218964 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "189a9347cd5d4b5f955e5c0706aa3947" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.219132 769 raft_consensus.cc:3058] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.220630 1170 ts_tablet_manager.cc:1397] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.005s
I20250625 13:52:22.221846 769 raft_consensus.cc:2466] T 189a9347cd5d4b5f955e5c0706aa3947 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.222332 851 leader_election.cc:304] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.221407 1170 raft_consensus.cc:357] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.222604 2573 raft_consensus.cc:2802] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.222649 1170 raft_consensus.cc:383] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.222741 1170 raft_consensus.cc:738] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.222719 2573 raft_consensus.cc:695] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.222954 1170 consensus_queue.cc:260] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.222975 2573 consensus_queue.cc:237] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.217872 2774 leader_election.cc:290] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.224090 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "189a9347cd5d4b5f955e5c0706aa3947" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.224511 1170 ts_tablet_manager.cc:1428] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 13:52:22.224817 1170 tablet_bootstrap.cc:492] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:22.225430 849 leader_election.cc:343] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.226406 2838 leader_election.cc:290] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.217859 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48645c913fc24c03a922cf6ea7531022" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.228327 711 leader_election.cc:343] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.229235 1170 tablet_bootstrap.cc:654] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.238349 1170 tablet_bootstrap.cc:492] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:22.238762 985 consensus_peers.cc:487] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 829221b8a79f460d90dd24be6a2d4c43. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.241098 1170 ts_tablet_manager.cc:1397] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250625 13:52:22.242123 1170 raft_consensus.cc:357] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.243461 1170 raft_consensus.cc:383] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.243753 1170 raft_consensus.cc:738] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
W20250625 13:52:22.247979 983 consensus_peers.cc:487] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:22.253400 851 consensus_peers.cc:487] T 4d2d6f05348e43339e7167a9cddca2ba P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d2d6f05348e43339e7167a9cddca2ba. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.259254 2774 raft_consensus.cc:491] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.259505 2774 raft_consensus.cc:513] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.260632 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535812d46d6444fda85d9767c6ca7e62" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.261899 851 leader_election.cc:343] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 535812d46d6444fda85d9767c6ca7e62
I20250625 13:52:22.262105 2774 leader_election.cc:290] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.264669 1170 consensus_queue.cc:260] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.265750 1048 raft_consensus.cc:2466] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.266371 849 leader_election.cc:304] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.266793 2573 raft_consensus.cc:2802] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.266863 2573 raft_consensus.cc:491] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.266913 2573 raft_consensus.cc:3058] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.267514 2774 raft_consensus.cc:491] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.267706 2774 raft_consensus.cc:513] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.268406 2774 leader_election.cc:290] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.269896 2573 raft_consensus.cc:513] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.270565 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cab4329d58404014833ae5029f8f296d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.270767 769 raft_consensus.cc:2466] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.272712 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cab4329d58404014833ae5029f8f296d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.272215 851 leader_election.cc:304] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:22.273746 849 leader_election.cc:343] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.275103 851 leader_election.cc:343] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 535812d46d6444fda85d9767c6ca7e62
I20250625 13:52:22.275332 1170 ts_tablet_manager.cc:1428] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20250625 13:52:22.275708 1170 tablet_bootstrap.cc:492] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.275740 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "535812d46d6444fda85d9767c6ca7e62" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.275894 1048 raft_consensus.cc:3058] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.276439 2573 leader_election.cc:290] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.276682 2573 raft_consensus.cc:2802] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.276829 2573 raft_consensus.cc:491] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.276955 2573 raft_consensus.cc:3058] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.279793 2636 raft_consensus.cc:491] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.279932 2636 raft_consensus.cc:513] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.280515 2636 leader_election.cc:290] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.281172 1170 tablet_bootstrap.cc:654] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.281366 2594 raft_consensus.cc:491] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.281515 2594 raft_consensus.cc:513] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.282114 2594 leader_election.cc:290] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.282934 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0012e95ff4924723a6e6aaf6b85da438" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:22.283422 711 leader_election.cc:343] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.284870 985 leader_election.cc:343] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9be75d03121f4ea2906236ac3a5b0764
I20250625 13:52:22.285526 1048 raft_consensus.cc:2466] T 535812d46d6444fda85d9767c6ca7e62 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.285943 849 leader_election.cc:304] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.286214 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9be75d03121f4ea2906236ac3a5b0764" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.286401 914 raft_consensus.cc:2466] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:22.286464 2838 raft_consensus.cc:2802] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.286623 2838 raft_consensus.cc:695] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.286739 983 leader_election.cc:304] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.286928 2838 consensus_queue.cc:237] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.287143 2636 raft_consensus.cc:2802] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.287256 2636 raft_consensus.cc:491] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.287359 2636 raft_consensus.cc:3058] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.289418 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0012e95ff4924723a6e6aaf6b85da438" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.289878 711 leader_election.cc:343] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.290004 711 leader_election.cc:304] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.289960 2636 raft_consensus.cc:513] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.290314 1168 ts_tablet_manager.cc:1918] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:22.290377 2594 raft_consensus.cc:2747] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:22.291599 985 leader_election.cc:343] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 9be75d03121f4ea2906236ac3a5b0764
I20250625 13:52:22.292156 2636 leader_election.cc:290] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.293102 1166 tablet_bootstrap.cc:492] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.293304 1166 ts_tablet_manager.cc:1397] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.002s
I20250625 13:52:22.294142 1166 raft_consensus.cc:357] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.294379 1166 raft_consensus.cc:383] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.294493 1166 raft_consensus.cc:738] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.294800 1166 consensus_queue.cc:260] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.295413 1166 ts_tablet_manager.cc:1428] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:22.297854 1166 tablet_bootstrap.cc:492] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.299487 1168 log.cc:1199] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/7506cce0ec4c4ec68f25591cb98b1252
I20250625 13:52:22.299857 1168 ts_tablet_manager.cc:1939] T 7506cce0ec4c4ec68f25591cb98b1252 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.300104 1166 tablet_bootstrap.cc:654] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.300623 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1429) took 2343 ms. Trace:
I20250625 13:52:22.300737 1168 rpcz_store.cc:276] 0625 13:52:19.957258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.960711 (+ 3453us) service_pool.cc:225] Handling call
0625 13:52:22.162163 (+2201452us) ts_tablet_manager.cc:1145] Deleting tablet 7506cce0ec4c4ec68f25591cb98b1252
0625 13:52:22.162176 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.196042 (+ 33866us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.290172 (+ 94130us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.300585 (+ 10413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2201277}
I20250625 13:52:22.301060 1168 tablet_replica.cc:331] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:22.301288 1168 raft_consensus.cc:2241] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:22.301613 1168 raft_consensus.cc:2270] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 13:52:22.302122 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1429) could send its response
I20250625 13:52:22.302513 1168 ts_tablet_manager.cc:1905] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.208601 1171 tablet_bootstrap.cc:654] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:22.306869 2747 op_tracker.cc:284] OpTracker waiting for 228 outstanding ops to complete now for 1.030s
I20250625 13:52:22.307008 2747 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20250625 13:52:22.307087 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016e700 [op_id=(term: 1 index: 1849), ts=7171520678107430912, rows=[]]]
I20250625 13:52:22.307427 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016e380 [op_id=(term: 1 index: 1848), ts=7171520677839859712, rows=[]]]
I20250625 13:52:22.307583 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016e000 [op_id=(term: 1 index: 1847), ts=7171520677820846080, rows=[]]]
I20250625 13:52:22.307684 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016dc80 [op_id=(term: 1 index: 1846), ts=7171520677794234368, rows=[]]]
I20250625 13:52:22.307797 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016d900 [op_id=(term: 1 index: 1845), ts=7171520677788327936, rows=[]]]
I20250625 13:52:22.307891 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016d580 [op_id=(term: 1 index: 1844), ts=7171520677776457728, rows=[]]]
I20250625 13:52:22.307986 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016d200 [op_id=(term: 1 index: 1843), ts=7171520677625921536, rows=[]]]
I20250625 13:52:22.308054 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016ce80 [op_id=(term: 1 index: 1842), ts=7171520677620477952, rows=[]]]
I20250625 13:52:22.308137 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016cb00 [op_id=(term: 1 index: 1841), ts=7171520677566840832, rows=[]]]
I20250625 13:52:22.308210 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016c780 [op_id=(term: 1 index: 1840), ts=7171520677013270528, rows=[]]]
I20250625 13:52:22.308301 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016c400 [op_id=(term: 1 index: 1839), ts=7171520677010313216, rows=[]]]
I20250625 13:52:22.308377 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016c080 [op_id=(term: 1 index: 1838), ts=7171520677007101952, rows=[]]]
I20250625 13:52:22.308490 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016bd00 [op_id=(term: 1 index: 1837), ts=7171520676984156160, rows=[]]]
I20250625 13:52:22.308579 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016b980 [op_id=(term: 1 index: 1836), ts=7171520676979318784, rows=[]]]
I20250625 13:52:22.308655 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016b600 [op_id=(term: 1 index: 1835), ts=7171520676976205824, rows=[]]]
I20250625 13:52:22.308741 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016b280 [op_id=(term: 1 index: 1834), ts=7171520676904214528, rows=[]]]
I20250625 13:52:22.308835 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016af00 [op_id=(term: 1 index: 1833), ts=7171520676893736960, rows=[]]]
I20250625 13:52:22.308907 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016ab80 [op_id=(term: 1 index: 1832), ts=7171520676812824576, rows=[]]]
I20250625 13:52:22.308984 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016a800 [op_id=(term: 1 index: 1831), ts=7171520676539416576, rows=[]]]
I20250625 13:52:22.290961 2573 raft_consensus.cc:513] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.309083 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016a480 [op_id=(term: 1 index: 1830), ts=7171520676399910912, rows=[]]]
I20250625 13:52:22.312160 2573 leader_election.cc:290] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.312297 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x61700016a100 [op_id=(term: 1 index: 1829), ts=7171520676397756416, rows=[]]]
I20250625 13:52:22.312469 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000169d80 [op_id=(term: 1 index: 1828), ts=7171520676377243648, rows=[]]]
I20250625 13:52:22.312721 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000169a00 [op_id=(term: 1 index: 1827), ts=7171520676309843968, rows=[]]]
I20250625 13:52:22.312891 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000169680 [op_id=(term: 1 index: 1826), ts=7171520676265242624, rows=[]]]
I20250625 13:52:22.313062 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000169300 [op_id=(term: 1 index: 1825), ts=7171520676256776192, rows=[]]]
I20250625 13:52:22.313226 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000168f80 [op_id=(term: 1 index: 1824), ts=7171520676125024256, rows=[]]]
I20250625 13:52:22.313417 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000168c00 [op_id=(term: 1 index: 1823), ts=7171520676122677248, rows=[]]]
I20250625 13:52:22.313580 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000168880 [op_id=(term: 1 index: 1822), ts=7171520676120141824, rows=[]]]
I20250625 13:52:22.292413 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9be75d03121f4ea2906236ac3a5b0764" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.313756 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000168500 [op_id=(term: 1 index: 1821), ts=7171520676117471232, rows=[]]]
I20250625 13:52:22.313859 914 raft_consensus.cc:3058] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.313865 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000168180 [op_id=(term: 1 index: 1820), ts=7171520676100231168, rows=[]]]
I20250625 13:52:22.313980 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000167e00 [op_id=(term: 1 index: 1819), ts=7171520676016242688, rows=[]]]
I20250625 13:52:22.314107 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000167a80 [op_id=(term: 1 index: 1818), ts=7171520676012625920, rows=[]]]
I20250625 13:52:22.314239 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000167700 [op_id=(term: 1 index: 1817), ts=7171520676005298176, rows=[]]]
I20250625 13:52:22.314352 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000167380 [op_id=(term: 1 index: 1816), ts=7171520676003389440, rows=[]]]
I20250625 13:52:22.314463 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000167000 [op_id=(term: 1 index: 1815), ts=7171520675993964544, rows=[]]]
I20250625 13:52:22.314568 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000166c80 [op_id=(term: 1 index: 1814), ts=7171520675992023040, rows=[]]]
I20250625 13:52:22.314692 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000166900 [op_id=(term: 1 index: 1813), ts=7171520675989983232, rows=[]]]
I20250625 13:52:22.314813 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000166580 [op_id=(term: 1 index: 1812), ts=7171520675987693568, rows=[]]]
I20250625 13:52:22.314929 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000165e80 [op_id=(term: 1 index: 1811), ts=7171520675948466176, rows=[]]]
I20250625 13:52:22.315037 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:20, state=WriteOpState 0x617000165b00 [op_id=(term: 1 index: 1810), ts=7171520675916062720, rows=[]]]
I20250625 13:52:22.315160 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000165780 [op_id=(term: 1 index: 1809), ts=7171520675544383488, rows=[]]]
I20250625 13:52:22.315260 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000165400 [op_id=(term: 1 index: 1808), ts=7171520675471384576, rows=[]]]
I20250625 13:52:22.315404 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000165080 [op_id=(term: 1 index: 1807), ts=7171520675447635968, rows=[]]]
I20250625 13:52:22.315518 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000164d00 [op_id=(term: 1 index: 1806), ts=7171520675444035584, rows=[]]]
I20250625 13:52:22.315659 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000164980 [op_id=(term: 1 index: 1805), ts=7171520675433574400, rows=[]]]
I20250625 13:52:22.315759 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000164600 [op_id=(term: 1 index: 1804), ts=7171520675228061696, rows=[]]]
I20250625 13:52:22.315876 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000164280 [op_id=(term: 1 index: 1803), ts=7171520675211804672, rows=[]]]
I20250625 13:52:22.315975 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000163f00 [op_id=(term: 1 index: 1802), ts=7171520675208581120, rows=[]]]
I20250625 13:52:22.316099 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000163b80 [op_id=(term: 1 index: 1801), ts=7171520675196510208, rows=[]]]
I20250625 13:52:22.316207 2747 op_tracker.cc:290] RD-NP WriteOp [type=FOLLOWER, start_time=2025-06-25 13:52:19, state=WriteOpState 0x617000163800 [op_id=(term: 1 index: 1800), ts=7171520675192881152, rows=[]]]
I20250625 13:52:22.316628 914 raft_consensus.cc:2466] T 9be75d03121f4ea2906236ac3a5b0764 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.317174 1170 tablet_bootstrap.cc:492] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.317353 1170 ts_tablet_manager.cc:1397] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.003s
I20250625 13:52:22.317982 983 leader_election.cc:304] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.318132 1170 raft_consensus.cc:357] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.318320 1170 raft_consensus.cc:383] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.318418 1170 raft_consensus.cc:738] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.318490 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cab4329d58404014833ae5029f8f296d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.318662 769 raft_consensus.cc:3058] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.318666 1170 consensus_queue.cc:260] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.318943 2638 raft_consensus.cc:2802] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.319059 2638 raft_consensus.cc:695] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.319298 2638 consensus_queue.cc:237] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.320266 2594 raft_consensus.cc:491] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.320379 2594 raft_consensus.cc:513] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.321117 2594 leader_election.cc:290] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:22.321352 769 raft_consensus.cc:2466] T cab4329d58404014833ae5029f8f296d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.321815 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35aedb3b590d4917bac8fe0962d68fb3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.322295 711 leader_election.cc:343] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.323019 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cab4329d58404014833ae5029f8f296d" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.331044 1170 ts_tablet_manager.cc:1428] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250625 13:52:22.332649 2594 raft_consensus.cc:491] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.332852 2594 raft_consensus.cc:513] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.333446 2594 leader_election.cc:290] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.334196 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b3d5a2ab7042c2a2f18890363e7a9b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.334426 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b3d5a2ab7042c2a2f18890363e7a9b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:22.335289 711 leader_election.cc:343] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.335510 711 leader_election.cc:343] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.335636 1166 tablet_bootstrap.cc:492] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.335583 711 leader_election.cc:304] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.335748 1166 ts_tablet_manager.cc:1397] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250625 13:52:22.336073 2594 raft_consensus.cc:2747] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.336545 1166 raft_consensus.cc:357] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.336715 1166 raft_consensus.cc:383] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.336768 1166 raft_consensus.cc:738] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.336966 1166 consensus_queue.cc:260] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.337477 1166 ts_tablet_manager.cc:1428] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:22.349068 2693 raft_consensus.cc:513] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.348163 1170 tablet_bootstrap.cc:492] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.350212 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48645c913fc24c03a922cf6ea7531022" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.350399 2693 leader_election.cc:290] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.350425 1046 raft_consensus.cc:3058] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.351684 1170 tablet_bootstrap.cc:654] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.352676 851 leader_election.cc:304] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.353286 1046 raft_consensus.cc:2466] T 48645c913fc24c03a922cf6ea7531022 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:22.353888 711 leader_election.cc:304] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:52:22.354105 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35aedb3b590d4917bac8fe0962d68fb3" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.354405 2693 raft_consensus.cc:2802] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.354542 2693 raft_consensus.cc:695] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:22.354805 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48645c913fc24c03a922cf6ea7531022" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.354780 2693 consensus_queue.cc:237] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.354969 915 raft_consensus.cc:3058] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.355386 1171 tablet_bootstrap.cc:492] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
W20250625 13:52:22.355598 849 leader_election.cc:343] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.356282 2838 raft_consensus.cc:2802] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.356421 2838 raft_consensus.cc:695] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.356595 1166 tablet_bootstrap.cc:492] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.356674 2838 consensus_queue.cc:237] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.357380 1171 ts_tablet_manager.cc:1397] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.151s user 0.003s sys 0.002s
I20250625 13:52:22.358585 2837 raft_consensus.cc:491] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.358778 2837 raft_consensus.cc:513] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.358860 2573 consensus_queue.cc:1035] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.359081 915 raft_consensus.cc:2466] T 48645c913fc24c03a922cf6ea7531022 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:22.359273 1166 tablet_bootstrap.cc:654] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.359817 2837 leader_election.cc:290] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:22.359462 711 leader_election.cc:343] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.361029 711 leader_election.cc:304] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.361470 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.361620 2594 raft_consensus.cc:2747] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.361644 778 raft_consensus.cc:2466] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.362725 851 leader_election.cc:304] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.362982 2573 raft_consensus.cc:2802] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.363068 2573 raft_consensus.cc:491] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.363134 2573 raft_consensus.cc:3058] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:22.363960 985 consensus_peers.cc:487] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d26820eaba98401da3a8b5348be21fc6. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.365732 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.365748 2573 raft_consensus.cc:513] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.366262 849 leader_election.cc:343] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.366375 2573 leader_election.cc:290] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.367013 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:22.367688 849 leader_election.cc:343] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.369330 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.369522 769 raft_consensus.cc:3058] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.373129 1171 raft_consensus.cc:357] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.373914 1171 raft_consensus.cc:383] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.374178 1171 raft_consensus.cc:738] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.374557 1171 consensus_queue.cc:260] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.376740 1168 ts_tablet_manager.cc:1918] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 13:52:22.378369 1168 log.cc:1199] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/50786a4c25274ffb8f9910c5dafe3933
I20250625 13:52:22.378929 1168 ts_tablet_manager.cc:1939] T 50786a4c25274ffb8f9910c5dafe3933 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.379335 1170 tablet_bootstrap.cc:492] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
W20250625 13:52:22.380069 983 consensus_peers.cc:487] T d26820eaba98401da3a8b5348be21fc6 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d26820eaba98401da3a8b5348be21fc6. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.380462 1170 ts_tablet_manager.cc:1397] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.001s
I20250625 13:52:22.381580 1170 raft_consensus.cc:357] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.383280 769 raft_consensus.cc:2466] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.383810 851 leader_election.cc:304] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.384099 2573 raft_consensus.cc:2802] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.384197 1166 tablet_bootstrap.cc:492] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.384356 1166 ts_tablet_manager.cc:1397] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250625 13:52:22.376832 1171 ts_tablet_manager.cc:1428] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250625 13:52:22.385005 1171 tablet_bootstrap.cc:492] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.385138 1166 raft_consensus.cc:357] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.385317 1166 raft_consensus.cc:383] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.385432 1166 raft_consensus.cc:738] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.385857 1166 consensus_queue.cc:260] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.386225 2573 raft_consensus.cc:695] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.386562 1166 ts_tablet_manager.cc:1428] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:22.386543 2573 consensus_queue.cc:237] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.386824 1166 tablet_bootstrap.cc:492] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.387090 1171 tablet_bootstrap.cc:654] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.388576 1166 tablet_bootstrap.cc:654] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.390076 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1428) took 2460 ms. Trace:
I20250625 13:52:22.390213 1168 rpcz_store.cc:276] 0625 13:52:19.929906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.929960 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:22.301002 (+2371042us) ts_tablet_manager.cc:1145] Deleting tablet 50786a4c25274ffb8f9910c5dafe3933
0625 13:52:22.301011 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.345561 (+ 44550us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.376565 (+ 31004us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.390040 (+ 13475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2335846}
W20250625 13:52:22.390640 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1428) could send its response
I20250625 13:52:22.390707 1168 tablet_replica.cc:331] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:22.390928 1168 raft_consensus.cc:2241] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:22.391153 1168 raft_consensus.cc:2270] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:22.391722 1168 ts_tablet_manager.cc:1905] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.382719 2636 raft_consensus.cc:491] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.398710 1170 raft_consensus.cc:383] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.398849 1170 raft_consensus.cc:738] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.398751 2636 raft_consensus.cc:513] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.399535 2636 leader_election.cc:290] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.400074 1170 consensus_queue.cc:260] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.400583 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "553af7c7326240838f37e3861d4e3357" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.400908 915 raft_consensus.cc:2466] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
W20250625 13:52:22.400959 985 leader_election.cc:343] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 553af7c7326240838f37e3861d4e3357
I20250625 13:52:22.404224 983 leader_election.cc:304] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.404500 2636 raft_consensus.cc:2802] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.404572 2636 raft_consensus.cc:491] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.404644 2636 raft_consensus.cc:3058] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.405089 1170 ts_tablet_manager.cc:1428] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250625 13:52:22.407732 2636 raft_consensus.cc:513] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.409343 2775 consensus_queue.cc:1035] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:52:22.412779 985 leader_election.cc:343] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 553af7c7326240838f37e3861d4e3357
I20250625 13:52:22.413484 2636 leader_election.cc:290] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.418121 1170 tablet_bootstrap.cc:492] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.424551 2866 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fec9d8a79054a5991ae90d8a3101e32)
I20250625 13:52:22.427129 1170 tablet_bootstrap.cc:654] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.427423 2866 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:22.430075 2091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58351267767f4d9a85f974ebc40db6fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 1850 } } ignore_live_leader: false dest_uuid: "3fec9d8a79054a5991ae90d8a3101e32" is_pre_election: true
I20250625 13:52:22.434480 2866 leader_election.cc:290] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
I20250625 13:52:22.432307 1046 raft_consensus.cc:3058] T 7d51b886302149839efc6f547b944639 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.440325 2774 consensus_queue.cc:1035] T 7d51b886302149839efc6f547b944639 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 13:52:22.444106 985 consensus_peers.cc:487] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e3ef7f5b474466bbc19fd31c6b6d99c. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:22.445123 513 leader_election.cc:336] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:52:22.445356 513 leader_election.cc:304] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58351267767f4d9a85f974ebc40db6fe; no voters: 3fec9d8a79054a5991ae90d8a3101e32, d85834f366fa42c294c9290af21ea111
I20250625 13:52:22.445699 2866 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250625 13:52:22.446576 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "553af7c7326240838f37e3861d4e3357" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.447227 915 raft_consensus.cc:3058] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.447249 2858 raft_consensus.cc:491] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.447412 2858 raft_consensus.cc:513] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.449404 1046 raft_consensus.cc:3058] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.449961 915 raft_consensus.cc:2466] T 553af7c7326240838f37e3861d4e3357 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.450840 983 leader_election.cc:304] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.452792 2638 raft_consensus.cc:2802] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.452960 2638 raft_consensus.cc:695] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.453240 2638 consensus_queue.cc:237] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.455152 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1e362b69fe40c0b6472b56f8cb9bcb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:22.455718 711 leader_election.cc:343] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.456925 2631 consensus_queue.cc:1035] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.457235 2858 leader_election.cc:290] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.457759 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1e362b69fe40c0b6472b56f8cb9bcb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.460085 711 leader_election.cc:343] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.460060 2631 consensus_queue.cc:1035] T 71cf3b95139245a4ba11b6bcaa3ab3f4 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.460179 711 leader_election.cc:304] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.460541 2693 raft_consensus.cc:2747] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.464282 1171 tablet_bootstrap.cc:492] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.464494 1171 ts_tablet_manager.cc:1397] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.003s
I20250625 13:52:22.465298 1171 raft_consensus.cc:357] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.465528 1171 raft_consensus.cc:383] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.465679 1171 raft_consensus.cc:738] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.466009 1171 consensus_queue.cc:260] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.467831 1171 ts_tablet_manager.cc:1428] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:22.469818 1171 tablet_bootstrap.cc:492] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.471590 1171 tablet_bootstrap.cc:654] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.473552 2638 raft_consensus.cc:491] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.473935 2638 raft_consensus.cc:513] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.475047 2638 leader_election.cc:290] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:22.477288 985 leader_election.cc:343] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 69c44d72516e4883aff636ca65929c11
I20250625 13:52:22.475486 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69c44d72516e4883aff636ca65929c11" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.478345 915 raft_consensus.cc:2466] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
W20250625 13:52:22.476877 849 consensus_peers.cc:487] T bd036ec08f774f5393067ce20bb07610 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.479029 1168 ts_tablet_manager.cc:1918] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:22.479149 983 leader_election.cc:304] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.483445 1168 log.cc:1199] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/1ca3f3760e6e41a29d004e6afe63ec98
I20250625 13:52:22.483453 2638 raft_consensus.cc:2802] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.483573 2638 raft_consensus.cc:491] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.483688 2638 raft_consensus.cc:3058] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.484228 1168 ts_tablet_manager.cc:1939] T 1ca3f3760e6e41a29d004e6afe63ec98 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.486541 2638 raft_consensus.cc:513] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.487633 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69c44d72516e4883aff636ca65929c11" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.487813 915 raft_consensus.cc:3058] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:22.487829 985 leader_election.cc:343] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 69c44d72516e4883aff636ca65929c11
I20250625 13:52:22.490448 915 raft_consensus.cc:2466] T 69c44d72516e4883aff636ca65929c11 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.490945 983 leader_election.cc:304] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.491423 2636 raft_consensus.cc:2802] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.492861 2636 raft_consensus.cc:695] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.493685 2638 leader_election.cc:290] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.493247 2636 consensus_queue.cc:237] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.500532 1166 tablet_bootstrap.cc:492] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.500686 1171 tablet_bootstrap.cc:492] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.500905 1171 ts_tablet_manager.cc:1397] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20250625 13:52:22.501142 2878 tablet.cc:1722] T 9fa52e79e00c4f9c97a05c353e11ae2c P 1c008de91eb444f48da2c4f0b2d7a1ea: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 13:52:22.501673 1171 raft_consensus.cc:357] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.501962 1171 raft_consensus.cc:383] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.502069 1171 raft_consensus.cc:738] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.502326 1171 consensus_queue.cc:260] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.502655 1166 ts_tablet_manager.cc:1397] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.116s user 0.001s sys 0.008s
I20250625 13:52:22.502911 1171 ts_tablet_manager.cc:1428] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:22.503832 1171 tablet_bootstrap.cc:492] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.503855 1166 raft_consensus.cc:357] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.508406 1170 tablet_bootstrap.cc:492] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.508626 1170 ts_tablet_manager.cc:1397] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.004s
I20250625 13:52:22.509585 1170 raft_consensus.cc:357] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.509845 1166 raft_consensus.cc:383] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.509960 1166 raft_consensus.cc:738] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.510241 1166 consensus_queue.cc:260] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.512475 1170 raft_consensus.cc:383] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.513373 1171 tablet_bootstrap.cc:654] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:22.514565 851 consensus_peers.cc:487] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef2c3757c0284aaca70ec317c05195d6. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.517606 1166 ts_tablet_manager.cc:1428] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20250625 13:52:22.517766 1170 raft_consensus.cc:738] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.507782 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1430) took 2525 ms. Trace:
I20250625 13:52:22.518144 1168 rpcz_store.cc:276] 0625 13:52:19.982588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:19.982646 (+ 58us) service_pool.cc:225] Handling call
0625 13:52:22.390633 (+2407987us) ts_tablet_manager.cc:1145] Deleting tablet 1ca3f3760e6e41a29d004e6afe63ec98
0625 13:52:22.390643 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.423665 (+ 33022us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.477925 (+ 54260us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.507743 (+ 29818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407704}
W20250625 13:52:22.518569 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1430) could send its response
I20250625 13:52:22.518711 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1431) took 2485 ms. Trace:
I20250625 13:52:22.518805 1168 rpcz_store.cc:276] 0625 13:52:20.033503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.033571 (+ 68us) service_pool.cc:225] Handling call
0625 13:52:22.518567 (+2484996us) ts_tablet_manager.cc:1145] Deleting tablet aa555aa6a0124e5bac31c2bb83bb26ae
0625 13:52:22.518575 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.518660 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2484812}
I20250625 13:52:22.518893 1170 consensus_queue.cc:260] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.519039 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1431) could send its response
I20250625 13:52:22.519145 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1432) took 2466 ms. Trace:
I20250625 13:52:22.519227 1168 rpcz_store.cc:276] 0625 13:52:20.052424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.052635 (+ 211us) service_pool.cc:225] Handling call
0625 13:52:22.519020 (+2466385us) ts_tablet_manager.cc:1145] Deleting tablet 78c79a2f6544493ba3f3ec20ea67fa95
0625 13:52:22.519033 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.519106 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2466156}
W20250625 13:52:22.519490 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1432) could send its response
I20250625 13:52:22.519580 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1434) took 2465 ms. Trace:
I20250625 13:52:22.519656 1168 rpcz_store.cc:276] 0625 13:52:20.054237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.054291 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:22.519462 (+2465171us) ts_tablet_manager.cc:1145] Deleting tablet 5af2b974df3a41f7b60537c9d8eac1e6
0625 13:52:22.519471 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.519538 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2464996}
W20250625 13:52:22.519896 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1434) could send its response
I20250625 13:52:22.519981 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1433) took 2467 ms. Trace:
I20250625 13:52:22.520072 1168 rpcz_store.cc:276] 0625 13:52:20.052576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.064433 (+ 11857us) service_pool.cc:225] Handling call
0625 13:52:22.519874 (+2455441us) ts_tablet_manager.cc:1145] Deleting tablet d12f07f4145240958d7decf21c03b6bc
0625 13:52:22.519885 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.519946 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2455224}
W20250625 13:52:22.520298 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1433) could send its response
I20250625 13:52:22.520344 1168 tablet_replica.cc:331] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:22.520555 1168 raft_consensus.cc:2241] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:22.520718 1168 raft_consensus.cc:2270] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:22.520992 1166 tablet_bootstrap.cc:492] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.521606 1170 ts_tablet_manager.cc:1428] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 13:52:22.521919 1170 tablet_bootstrap.cc:492] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.523602 1170 tablet_bootstrap.cc:654] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.536356 1166 tablet_bootstrap.cc:654] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:22.527211 849 consensus_peers.cc:487] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c49ef682cbb46e1ae5c3b91e4e2d9c9. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.540022 1168 ts_tablet_manager.cc:1905] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.524317 2638 raft_consensus.cc:491] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.541401 2638 raft_consensus.cc:513] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.542093 2638 leader_election.cc:290] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.542141 1171 tablet_bootstrap.cc:492] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.525434 2659 raft_consensus.cc:491] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.542286 1171 ts_tablet_manager.cc:1397] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.008s
I20250625 13:52:22.542269 2659 raft_consensus.cc:513] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.542769 2659 leader_election.cc:290] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.542969 1171 raft_consensus.cc:357] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.543133 1171 raft_consensus.cc:383] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.543223 1171 raft_consensus.cc:738] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.543454 1171 consensus_queue.cc:260] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.544030 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af704cdf44ef42f7b5ea10b199d0f017" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.544059 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7efc8df3c8e24020b073a1b61bc9e0a0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.544195 915 raft_consensus.cc:2466] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:22.544232 914 raft_consensus.cc:2466] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:22.545296 983 leader_election.cc:304] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.545545 983 leader_election.cc:304] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.545727 2638 raft_consensus.cc:2802] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.545809 2638 raft_consensus.cc:491] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.545840 2659 raft_consensus.cc:2802] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.545862 2638 raft_consensus.cc:3058] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.545917 2659 raft_consensus.cc:491] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.545969 2659 raft_consensus.cc:3058] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.548368 2659 raft_consensus.cc:513] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.548524 2638 raft_consensus.cc:513] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.548878 2659 leader_election.cc:290] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.549088 2638 leader_election.cc:290] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.550639 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af704cdf44ef42f7b5ea10b199d0f017" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.550812 915 raft_consensus.cc:3058] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.550998 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7efc8df3c8e24020b073a1b61bc9e0a0" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.551147 914 raft_consensus.cc:3058] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.553512 915 raft_consensus.cc:2466] T af704cdf44ef42f7b5ea10b199d0f017 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.553972 914 raft_consensus.cc:2466] T 7efc8df3c8e24020b073a1b61bc9e0a0 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.554015 983 leader_election.cc:304] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.554077 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af704cdf44ef42f7b5ea10b199d0f017" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.554250 764 raft_consensus.cc:2466] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:22.555925 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af704cdf44ef42f7b5ea10b199d0f017" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.556080 764 raft_consensus.cc:3058] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.556624 983 leader_election.cc:304] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.558516 764 raft_consensus.cc:2466] T af704cdf44ef42f7b5ea10b199d0f017 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.562045 1171 ts_tablet_manager.cc:1428] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250625 13:52:22.565469 1168 ts_tablet_manager.cc:1918] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:22.565596 1168 log.cc:1199] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/d9462da7574a46839763b1c388102e60
I20250625 13:52:22.565969 1168 ts_tablet_manager.cc:1939] T d9462da7574a46839763b1c388102e60 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.566771 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1435) took 2490 ms. Trace:
I20250625 13:52:22.566861 1168 rpcz_store.cc:276] 0625 13:52:20.076167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.076213 (+ 46us) service_pool.cc:225] Handling call
0625 13:52:22.520279 (+2444066us) ts_tablet_manager.cc:1145] Deleting tablet d9462da7574a46839763b1c388102e60
0625 13:52:22.520290 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.558134 (+ 37844us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.565352 (+ 7218us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.566693 (+ 1341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2443881}
I20250625 13:52:22.567283 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1436) took 2420 ms. Trace:
W20250625 13:52:22.567366 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1435) could send its response
I20250625 13:52:22.567399 1168 rpcz_store.cc:276] 0625 13:52:20.146474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.146528 (+ 54us) service_pool.cc:225] Handling call
0625 13:52:22.567110 (+2420582us) ts_tablet_manager.cc:1145] Deleting tablet d406206ea8e14121bf43214adfc5afda
0625 13:52:22.567119 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.567220 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2420385}
I20250625 13:52:22.567713 1168 tablet_replica.cc:331] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
W20250625 13:52:22.567801 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1436) could send its response
I20250625 13:52:22.567916 1168 raft_consensus.cc:2241] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:22.568077 1168 raft_consensus.cc:2270] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:22.568545 1168 ts_tablet_manager.cc:1905] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.570711 1171 tablet_bootstrap.cc:492] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.572469 1171 tablet_bootstrap.cc:654] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.581722 2638 raft_consensus.cc:2802] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.581877 2638 raft_consensus.cc:695] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.582125 2638 consensus_queue.cc:237] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.583472 2659 raft_consensus.cc:2802] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.583600 2659 raft_consensus.cc:695] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.583789 2659 consensus_queue.cc:237] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.585727 983 consensus_peers.cc:487] T 829221b8a79f460d90dd24be6a2d4c43 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 829221b8a79f460d90dd24be6a2d4c43. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.592298 1168 ts_tablet_manager.cc:1918] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:22.592451 1168 log.cc:1199] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/db6e9db2375144b38ec43ffca9ca4c1e
I20250625 13:52:22.592922 1168 ts_tablet_manager.cc:1939] T db6e9db2375144b38ec43ffca9ca4c1e P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.593817 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1437) took 2243 ms. Trace:
I20250625 13:52:22.593950 1168 rpcz_store.cc:276] 0625 13:52:20.350444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.350765 (+ 321us) service_pool.cc:225] Handling call
0625 13:52:22.567633 (+2216868us) ts_tablet_manager.cc:1145] Deleting tablet db6e9db2375144b38ec43ffca9ca4c1e
0625 13:52:22.567641 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.588153 (+ 20512us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.592114 (+ 3961us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.593770 (+ 1656us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216674}
W20250625 13:52:22.594246 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1437) could send its response
I20250625 13:52:22.600312 1171 tablet_bootstrap.cc:492] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
W20250625 13:52:22.600891 985 consensus_peers.cc:487] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eafc402d809245d189f3977162506ef8. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:22.601212 985 consensus_peers.cc:487] T 7506cce0ec4c4ec68f25591cb98b1252 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7506cce0ec4c4ec68f25591cb98b1252. This is attempt 1: this message will repeat every 5th retry.
W20250625 13:52:22.601634 985 leader_election.cc:343] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7efc8df3c8e24020b073a1b61bc9e0a0
W20250625 13:52:22.602000 985 leader_election.cc:343] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 7efc8df3c8e24020b073a1b61bc9e0a0
I20250625 13:52:22.604856 1166 tablet_bootstrap.cc:492] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.605042 1166 ts_tablet_manager.cc:1397] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.084s user 0.000s sys 0.004s
I20250625 13:52:22.607138 1170 tablet_bootstrap.cc:492] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.607333 1170 ts_tablet_manager.cc:1397] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20250625 13:52:22.608094 1170 raft_consensus.cc:357] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.608516 1170 raft_consensus.cc:383] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.611572 1171 ts_tablet_manager.cc:1397] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.018s
I20250625 13:52:22.612429 1171 raft_consensus.cc:357] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.612656 1171 raft_consensus.cc:383] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.612751 1171 raft_consensus.cc:738] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.613016 1171 consensus_queue.cc:260] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.605916 1166 raft_consensus.cc:357] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.613566 1171 ts_tablet_manager.cc:1428] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:22.613675 1168 tablet_replica.cc:331] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:22.613780 1166 raft_consensus.cc:383] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.613909 1168 raft_consensus.cc:2241] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Raft consensus shutting down.
I20250625 13:52:22.608762 2881 raft_consensus.cc:491] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.614100 2881 raft_consensus.cc:513] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.614470 1170 raft_consensus.cc:738] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.615167 2838 raft_consensus.cc:491] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.615343 2838 raft_consensus.cc:513] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.616156 2838 leader_election.cc:290] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.616982 1168 raft_consensus.cc:2270] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:22.617321 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0012e95ff4924723a6e6aaf6b85da438" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.617518 764 raft_consensus.cc:2466] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.617594 1168 ts_tablet_manager.cc:1905] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.618119 851 leader_election.cc:304] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.618808 2573 raft_consensus.cc:2802] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.618901 2573 raft_consensus.cc:491] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.619004 2573 raft_consensus.cc:3058] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.621543 2573 raft_consensus.cc:513] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.622159 2573 leader_election.cc:290] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.614639 1166 raft_consensus.cc:738] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.622421 1166 consensus_queue.cc:260] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.616986 2881 leader_election.cc:290] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.623013 1170 consensus_queue.cc:260] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.618693 983 leader_election.cc:343] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Not found: Tablet not found: 32a57268a18f4f00a39f8360f9c90737
I20250625 13:52:22.623260 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0012e95ff4924723a6e6aaf6b85da438" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:22.623751 985 leader_election.cc:343] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 32a57268a18f4f00a39f8360f9c90737
I20250625 13:52:22.624048 985 leader_election.cc:304] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.624323 2881 raft_consensus.cc:2747] T 32a57268a18f4f00a39f8360f9c90737 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.617065 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0012e95ff4924723a6e6aaf6b85da438" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.625909 849 leader_election.cc:343] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.626121 849 leader_election.cc:343] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.626901 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0012e95ff4924723a6e6aaf6b85da438" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:22.627086 774 raft_consensus.cc:3058] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.627581 1171 tablet_bootstrap.cc:492] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.630602 774 raft_consensus.cc:2466] T 0012e95ff4924723a6e6aaf6b85da438 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:22.631101 851 leader_election.cc:304] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.631388 2573 raft_consensus.cc:2802] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.631539 2573 raft_consensus.cc:695] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:22.631769 2573 consensus_queue.cc:237] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.634773 2858 raft_consensus.cc:491] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.634912 2858 raft_consensus.cc:513] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.635598 2858 leader_election.cc:290] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.637231 2858 raft_consensus.cc:491] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.637341 2858 raft_consensus.cc:513] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.637584 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29a542de77674e33b70647ffd9475fbf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.637785 2858 leader_election.cc:290] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:22.638126 711 leader_election.cc:343] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.638412 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29a542de77674e33b70647ffd9475fbf" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.638532 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e8b9b732f8948d2b4c1a81a1de4489a" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.638801 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e8b9b732f8948d2b4c1a81a1de4489a" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:22.639740 711 leader_election.cc:343] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.639837 711 leader_election.cc:304] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:22.640159 711 leader_election.cc:343] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.640334 711 leader_election.cc:343] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.640409 711 leader_election.cc:304] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.640751 2693 raft_consensus.cc:2747] T 29a542de77674e33b70647ffd9475fbf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.640900 2693 raft_consensus.cc:2747] T 6e8b9b732f8948d2b4c1a81a1de4489a P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.642726 1170 ts_tablet_manager.cc:1428] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20250625 13:52:22.643065 1166 ts_tablet_manager.cc:1428] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20250625 13:52:22.644279 1170 tablet_bootstrap.cc:492] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.645932 1170 tablet_bootstrap.cc:654] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.649572 1171 tablet_bootstrap.cc:654] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.657641 1166 tablet_bootstrap.cc:492] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.657634 1168 ts_tablet_manager.cc:1918] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 13:52:22.657797 1168 log.cc:1199] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/4c49ef682cbb46e1ae5c3b91e4e2d9c9
I20250625 13:52:22.658206 1168 ts_tablet_manager.cc:1939] T 4c49ef682cbb46e1ae5c3b91e4e2d9c9 P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.658996 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1438) took 2308 ms. Trace:
I20250625 13:52:22.659096 1168 rpcz_store.cc:276] 0625 13:52:20.350592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.351004 (+ 412us) service_pool.cc:225] Handling call
0625 13:52:22.611439 (+2260435us) ts_tablet_manager.cc:1145] Deleting tablet 4c49ef682cbb46e1ae5c3b91e4e2d9c9
0625 13:52:22.611454 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.637999 (+ 26545us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.657501 (+ 19502us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.658947 (+ 1446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2260287}
I20250625 13:52:22.659377 1166 tablet_bootstrap.cc:654] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
W20250625 13:52:22.667125 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1438) could send its response
I20250625 13:52:22.667675 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1439) took 2316 ms. Trace:
I20250625 13:52:22.667829 1168 rpcz_store.cc:276] 0625 13:52:20.350706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.351173 (+ 467us) service_pool.cc:225] Handling call
0625 13:52:22.667487 (+2316314us) ts_tablet_manager.cc:1145] Deleting tablet e29bade90b184d2483cb1fe85033a03c
0625 13:52:22.667497 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.667626 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2316187}
W20250625 13:52:22.668128 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1439) could send its response
I20250625 13:52:22.668428 1168 tablet_replica.cc:331] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: stopping tablet replica
I20250625 13:52:22.673532 1168 raft_consensus.cc:2241] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 13:52:22.674049 1168 raft_consensus.cc:2270] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 13:52:22.674855 1168 ts_tablet_manager.cc:1905] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 13:52:22.685220 1168 ts_tablet_manager.cc:1918] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 13:52:22.686697 1171 tablet_bootstrap.cc:492] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.691555 1168 log.cc:1199] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: Deleting WAL directory at /tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/ts-1/wal/wals/0e3ef7f5b474466bbc19fd31c6b6d99c
I20250625 13:52:22.691637 1171 ts_tablet_manager.cc:1397] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.002s
I20250625 13:52:22.691968 1168 ts_tablet_manager.cc:1939] T 0e3ef7f5b474466bbc19fd31c6b6d99c P affa1c3035b940e88ca143be7d35840d: Deleting consensus metadata
I20250625 13:52:22.692440 1171 raft_consensus.cc:357] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.692626 1171 raft_consensus.cc:383] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.692701 1171 raft_consensus.cc:738] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.692938 1171 consensus_queue.cc:260] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.693420 1171 ts_tablet_manager.cc:1428] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:22.693750 1171 tablet_bootstrap.cc:492] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.696100 1166 tablet_bootstrap.cc:492] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.696228 1166 ts_tablet_manager.cc:1397] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.003s
I20250625 13:52:22.696959 1166 raft_consensus.cc:357] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.697109 1166 raft_consensus.cc:383] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.697156 1166 raft_consensus.cc:738] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.697371 1166 consensus_queue.cc:260] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.698017 1166 ts_tablet_manager.cc:1428] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:22.698791 1171 tablet_bootstrap.cc:654] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.699759 1170 tablet_bootstrap.cc:492] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.699887 1170 ts_tablet_manager.cc:1397] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20250625 13:52:22.700578 1170 raft_consensus.cc:357] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.700743 1170 raft_consensus.cc:383] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.700801 1170 raft_consensus.cc:738] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.700992 1170 consensus_queue.cc:260] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.704488 1168 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1440) took 2235 ms. Trace:
I20250625 13:52:22.704654 1168 rpcz_store.cc:276] 0625 13:52:20.461040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 13:52:20.461105 (+ 65us) service_pool.cc:225] Handling call
0625 13:52:22.668351 (+2207246us) ts_tablet_manager.cc:1145] Deleting tablet 0e3ef7f5b474466bbc19fd31c6b6d99c
0625 13:52:22.668359 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 13:52:22.677960 (+ 9601us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.685073 (+ 7113us) tablet_metadata.cc:641] Metadata flushed
0625 13:52:22.696031 (+ 10958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2200025}
W20250625 13:52:22.704998 850 connection.cc:441] server connection from 127.0.0.1:52994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52994 (request call id 1440) could send its response
I20250625 13:52:22.719966 1171 tablet_bootstrap.cc:492] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.720218 1171 ts_tablet_manager.cc:1397] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.009s
I20250625 13:52:22.721347 1171 raft_consensus.cc:357] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.721639 1171 raft_consensus.cc:383] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.721609 2573 consensus_queue.cc:1035] T ef2c3757c0284aaca70ec317c05195d6 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.721869 1171 raft_consensus.cc:738] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.722107 1171 consensus_queue.cc:260] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.722535 983 consensus_peers.cc:487] T 0e3ef7f5b474466bbc19fd31c6b6d99c P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e3ef7f5b474466bbc19fd31c6b6d99c. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.722579 1171 ts_tablet_manager.cc:1428] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:22.726186 1171 tablet_bootstrap.cc:492] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.728422 1171 tablet_bootstrap.cc:654] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.737205 1166 tablet_bootstrap.cc:492] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:22.744100 983 consensus_peers.cc:487] T eafc402d809245d189f3977162506ef8 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eafc402d809245d189f3977162506ef8. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.746146 1166 tablet_bootstrap.cc:654] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.749857 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
/tmp/dist-test-taskWNKxCE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWNKxCE/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750859507785809-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43307
--webserver_interface=127.0.104.60
--webserver_port=37999
--builtin_ntp_servers=127.0.104.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33015,127.0.104.61:45727,127.0.104.60:43307
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 13:52:22.750852 1170 ts_tablet_manager.cc:1428] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20250625 13:52:22.753373 1170 tablet_bootstrap.cc:492] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.755417 1170 tablet_bootstrap.cc:654] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.760270 1170 tablet_bootstrap.cc:492] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:22.760396 1170 ts_tablet_manager.cc:1397] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250625 13:52:22.761320 1170 raft_consensus.cc:357] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.761489 1170 raft_consensus.cc:383] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.761544 1170 raft_consensus.cc:738] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.762521 1170 consensus_queue.cc:260] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.762975 1170 ts_tablet_manager.cc:1428] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:22.765167 2858 raft_consensus.cc:491] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.765303 2858 raft_consensus.cc:513] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.765865 2858 leader_election.cc:290] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.766688 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf7e30cdff245208dacc6963fe88c7d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.766710 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf7e30cdff245208dacc6963fe88c7d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.766916 914 raft_consensus.cc:2466] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
W20250625 13:52:22.767228 711 leader_election.cc:343] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.767472 711 leader_election.cc:304] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.767891 2858 raft_consensus.cc:2802] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.767961 2858 raft_consensus.cc:491] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.768019 2858 raft_consensus.cc:3058] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.770615 2858 raft_consensus.cc:513] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.780570 1171 tablet_bootstrap.cc:492] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.780750 1171 ts_tablet_manager.cc:1397] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.014s
I20250625 13:52:22.781497 1171 raft_consensus.cc:357] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.781675 1171 raft_consensus.cc:383] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.781764 1171 raft_consensus.cc:738] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.781978 1171 consensus_queue.cc:260] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.782466 1171 ts_tablet_manager.cc:1428] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 13:52:22.782843 1171 tablet_bootstrap.cc:492] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.784572 1171 tablet_bootstrap.cc:654] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.785874 2693 raft_consensus.cc:491] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.786031 2693 raft_consensus.cc:513] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.786739 2693 leader_election.cc:290] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.787911 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6d0dbbe2894e61b0ea08f98bce61e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.788460 2693 raft_consensus.cc:491] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.788585 2693 raft_consensus.cc:513] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.789090 2693 leader_election.cc:290] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
W20250625 13:52:22.789686 711 leader_election.cc:343] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.790293 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17b172c7631048ceb97ccbea896f7f5b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.790462 914 raft_consensus.cc:2466] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:22.790858 711 leader_election.cc:304] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.791121 2693 raft_consensus.cc:2802] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.791179 2693 raft_consensus.cc:491] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.791234 2693 raft_consensus.cc:3058] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.791292 1166 tablet_bootstrap.cc:492] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.791471 1166 ts_tablet_manager.cc:1397] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.030s
I20250625 13:52:22.792213 1166 raft_consensus.cc:357] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.792392 1166 raft_consensus.cc:383] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.792454 1166 raft_consensus.cc:738] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.794189 2693 raft_consensus.cc:513] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.794332 1166 consensus_queue.cc:260] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.807466 2858 leader_election.cc:290] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.808065 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf7e30cdff245208dacc6963fe88c7d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.808247 914 raft_consensus.cc:3058] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.811087 914 raft_consensus.cc:2466] T cbf7e30cdff245208dacc6963fe88c7d P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:22.812317 1171 tablet_bootstrap.cc:492] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.812521 1171 ts_tablet_manager.cc:1397] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20250625 13:52:22.813666 1171 raft_consensus.cc:357] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.814028 2659 raft_consensus.cc:491] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.814174 2659 raft_consensus.cc:513] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.814828 2659 leader_election.cc:290] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.816392 2693 leader_election.cc:290] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.818213 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.818421 774 raft_consensus.cc:2391] T 0a90fd8cb29a4fffa65129dbca9ad84c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:52:22.807660 1166 ts_tablet_manager.cc:1428] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 13:52:22.812749 711 leader_election.cc:304] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.819448 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a90fd8cb29a4fffa65129dbca9ad84c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.819525 2693 raft_consensus.cc:2802] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.819635 2693 raft_consensus.cc:695] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:22.819877 2693 consensus_queue.cc:237] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.821295 985 consensus_peers.cc:487] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db6e9db2375144b38ec43ffca9ca4c1e. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.821540 1166 tablet_bootstrap.cc:492] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.822919 983 leader_election.cc:304] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.823418 2659 raft_consensus.cc:3058] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.826323 2659 raft_consensus.cc:2747] T 0a90fd8cb29a4fffa65129dbca9ad84c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.819448 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17b172c7631048ceb97ccbea896f7f5b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.830324 1171 raft_consensus.cc:383] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.830569 1171 raft_consensus.cc:738] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.830855 1171 consensus_queue.cc:260] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.831485 1171 ts_tablet_manager.cc:1428] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250625 13:52:22.830487 914 raft_consensus.cc:3058] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.833009 2881 consensus_queue.cc:1035] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.837826 914 raft_consensus.cc:2466] T 17b172c7631048ceb97ccbea896f7f5b P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:22.840574 711 leader_election.cc:304] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:22.840847 2903 raft_consensus.cc:2802] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.841046 2903 raft_consensus.cc:695] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:22.841492 2903 consensus_queue.cc:237] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.841687 1171 tablet_bootstrap.cc:492] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.842746 2659 raft_consensus.cc:491] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.842937 2659 raft_consensus.cc:513] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:22.844290 985 leader_election.cc:343] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: e7bd4939714a4359846345ed0521701c
I20250625 13:52:22.844921 2659 leader_election.cc:290] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.845217 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7bd4939714a4359846345ed0521701c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:22.845469 914 raft_consensus.cc:2466] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
I20250625 13:52:22.845996 983 leader_election.cc:304] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.846398 2659 raft_consensus.cc:2802] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.846524 2659 raft_consensus.cc:491] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.846642 2659 raft_consensus.cc:3058] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.849272 2693 raft_consensus.cc:491] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.849334 2659 raft_consensus.cc:513] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.849443 2693 raft_consensus.cc:513] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.851055 2693 leader_election.cc:290] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:22.851356 985 leader_election.cc:343] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: e7bd4939714a4359846345ed0521701c
I20250625 13:52:22.851681 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efc9536a716f4b9f9fc4af1637b3cd80" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:22.852224 711 leader_election.cc:343] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.853204 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7bd4939714a4359846345ed0521701c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:22.853510 2659 leader_election.cc:290] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.853744 914 raft_consensus.cc:3058] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.854254 1166 tablet_bootstrap.cc:654] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.855077 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17b172c7631048ceb97ccbea896f7f5b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.860541 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf7e30cdff245208dacc6963fe88c7d" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.860913 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6d0dbbe2894e61b0ea08f98bce61e7" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.861207 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17b172c7631048ceb97ccbea896f7f5b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.861665 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efc9536a716f4b9f9fc4af1637b3cd80" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.875348 711 leader_election.cc:343] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.875689 711 leader_election.cc:343] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.876026 711 leader_election.cc:343] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.876121 711 leader_election.cc:304] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.876542 2903 raft_consensus.cc:2747] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 13:52:22.876936 711 leader_election.cc:343] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.877480 711 leader_election.cc:343] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.877880 711 leader_election.cc:304] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.878683 2903 raft_consensus.cc:2747] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.881409 1170 tablet_bootstrap.cc:492] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:22.887216 2747 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:52:22.888973 1170 tablet_bootstrap.cc:654] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.903865 1166 tablet_bootstrap.cc:492] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:22.904038 1166 ts_tablet_manager.cc:1397] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.000s
I20250625 13:52:22.904876 1166 raft_consensus.cc:357] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.905088 1166 raft_consensus.cc:383] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.905167 1166 raft_consensus.cc:738] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.905371 1166 consensus_queue.cc:260] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.905849 1166 ts_tablet_manager.cc:1428] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250625 13:52:22.906136 1166 tablet_bootstrap.cc:492] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:22.910307 1166 tablet_bootstrap.cc:654] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.914315 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_469944f606314307b97ad3e50d27ad30 [id=00360a43d057493bab6d55d317094943]
I20250625 13:52:22.914914 1171 tablet_bootstrap.cc:654] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.915213 2573 consensus_queue.cc:1035] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250625 13:52:22.929157 914 raft_consensus.cc:2466] T e7bd4939714a4359846345ed0521701c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:22.934193 983 leader_election.cc:304] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:22.934988 2903 raft_consensus.cc:491] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.935124 2903 raft_consensus.cc:513] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.935834 2903 leader_election.cc:290] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:22.936829 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e73bea55af9e4890bf7070b488313c3f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:22.937518 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e73bea55af9e4890bf7070b488313c3f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:22.939908 711 leader_election.cc:343] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:22.940090 711 leader_election.cc:343] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.940151 711 leader_election.cc:304] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:22.940613 2903 raft_consensus.cc:2747] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:22.941155 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_9c3c0c15b51d4dca8c18dfb78f2054d5 [id=02c7e2be40284b18af9258ccd83a7e8e]
I20250625 13:52:22.941905 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_c9f8713b891a4605a583946a0fa51b1e [id=031831da59e54c3aa1d464d71ad2d91a]
I20250625 13:52:22.942901 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_26b209437c764864aae2091be187dd11 [id=03b41b1e711a4af484ad35073f456efa]
I20250625 13:52:22.945598 2659 raft_consensus.cc:2802] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:22.945734 2659 raft_consensus.cc:695] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:22.945961 2659 consensus_queue.cc:237] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.954375 1050 raft_consensus.cc:3058] T 6f5780d88dea41e7b855b41f12e96c52 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.960371 2580 consensus_queue.cc:1035] T 6f5780d88dea41e7b855b41f12e96c52 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.968530 2580 consensus_queue.cc:1035] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:22.980751 1171 tablet_bootstrap.cc:492] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:22.980906 1171 ts_tablet_manager.cc:1397] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.004s
I20250625 13:52:22.981577 1171 raft_consensus.cc:357] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.981740 1171 raft_consensus.cc:383] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:22.981786 1171 raft_consensus.cc:738] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:22.981983 1171 consensus_queue.cc:260] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:22.983629 1171 ts_tablet_manager.cc:1428] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 13:52:22.985180 1171 tablet_bootstrap.cc:492] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:22.988178 1171 tablet_bootstrap.cc:654] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:22.989112 2580 raft_consensus.cc:491] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.989226 2580 raft_consensus.cc:513] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.989791 2580 leader_election.cc:290] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.990419 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb648c45ea344fb3bac765c9e06db2b1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.990947 849 leader_election.cc:343] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.991341 2580 raft_consensus.cc:491] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:22.991447 2580 raft_consensus.cc:513] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.991917 2580 leader_election.cc:290] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.992444 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1310fee26fe642bf9919a2bd8e1c091f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:22.993005 849 leader_election.cc:343] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:22.993592 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb648c45ea344fb3bac765c9e06db2b1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.993693 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1310fee26fe642bf9919a2bd8e1c091f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:22.993772 764 raft_consensus.cc:2466] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.993870 769 raft_consensus.cc:2466] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:22.994252 851 leader_election.cc:304] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.994643 851 leader_election.cc:304] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:22.994894 2775 raft_consensus.cc:2802] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.994967 2775 raft_consensus.cc:491] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.995005 2580 raft_consensus.cc:2802] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:22.995033 2775 raft_consensus.cc:3058] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.995066 2580 raft_consensus.cc:491] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:22.995131 2580 raft_consensus.cc:3058] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:22.997699 2580 raft_consensus.cc:513] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.997756 2775 raft_consensus.cc:513] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:22.998240 2775 leader_election.cc:290] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:22.998266 2580 leader_election.cc:290] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:22.998992 985 consensus_peers.cc:487] T 4ae5f1d414794474b2178b431cbc1d44 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ae5f1d414794474b2178b431cbc1d44. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:22.999028 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1310fee26fe642bf9919a2bd8e1c091f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:22.999339 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb648c45ea344fb3bac765c9e06db2b1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:22.999812 849 leader_election.cc:343] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:23.000015 849 leader_election.cc:343] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.001636 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1310fee26fe642bf9919a2bd8e1c091f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.001814 769 raft_consensus.cc:3058] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.005304 769 raft_consensus.cc:2466] T 1310fee26fe642bf9919a2bd8e1c091f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.006150 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb648c45ea344fb3bac765c9e06db2b1" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.006385 774 raft_consensus.cc:3058] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.006999 851 leader_election.cc:304] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.007387 2775 raft_consensus.cc:2802] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.007515 2775 raft_consensus.cc:695] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.007970 2775 consensus_queue.cc:237] T 1310fee26fe642bf9919a2bd8e1c091f P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.009521 774 raft_consensus.cc:2466] T bb648c45ea344fb3bac765c9e06db2b1 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.016911 1170 tablet_bootstrap.cc:492] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.017156 1170 ts_tablet_manager.cc:1397] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.136s user 0.015s sys 0.003s
I20250625 13:52:23.017299 851 leader_election.cc:304] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.017743 2573 raft_consensus.cc:2802] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.017866 2573 raft_consensus.cc:695] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.018095 2573 consensus_queue.cc:237] T bb648c45ea344fb3bac765c9e06db2b1 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:23.028702 985 consensus_peers.cc:487] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ca3f3760e6e41a29d004e6afe63ec98. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.034018 1170 raft_consensus.cc:357] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.034288 1170 raft_consensus.cc:383] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.034395 1170 raft_consensus.cc:738] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.034744 1170 consensus_queue.cc:260] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.035247 1170 ts_tablet_manager.cc:1428] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250625 13:52:23.035604 1170 tablet_bootstrap.cc:492] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
W20250625 13:52:23.037130 983 consensus_peers.cc:487] T 1ca3f3760e6e41a29d004e6afe63ec98 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ca3f3760e6e41a29d004e6afe63ec98. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.040977 2903 raft_consensus.cc:491] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.041126 2903 raft_consensus.cc:513] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.041754 2903 leader_election.cc:290] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.042781 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3016763b9e294b9da2861fe97e57e241" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:23.043458 711 leader_election.cc:343] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.042567 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3016763b9e294b9da2861fe97e57e241" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.044214 711 leader_election.cc:343] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.044344 711 leader_election.cc:304] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.044777 2903 raft_consensus.cc:2747] T 3016763b9e294b9da2861fe97e57e241 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.045518 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_167a0a50523949028824975d052ed256 [id=040cb0194b024efc8d97457be683b978]
I20250625 13:52:23.046906 1170 tablet_bootstrap.cc:654] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.056212 2573 consensus_queue.cc:1035] T 0a90fd8cb29a4fffa65129dbca9ad84c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.062381 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_2498f4d6086d4a1b98304771c890d55f [id=050b47e9354146958edc009630ea65d4]
I20250625 13:52:23.063006 2883 raft_consensus.cc:491] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.063186 2883 raft_consensus.cc:513] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.063932 2883 leader_election.cc:290] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.064697 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.064966 769 raft_consensus.cc:2391] T 663588b7e9ee430faa180fec00897e22 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:52:23.065500 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_4dc8c488c3e44f3f882049d1c0f8feb2 [id=0532faedda28481f9742008aecb6bfd3]
I20250625 13:52:23.066956 1166 tablet_bootstrap.cc:492] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.067180 1166 ts_tablet_manager.cc:1397] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.161s user 0.004s sys 0.059s
W20250625 13:52:23.067277 851 consensus_peers.cc:487] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 535812d46d6444fda85d9767c6ca7e62. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.067970 1171 tablet_bootstrap.cc:492] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:23.068111 1171 ts_tablet_manager.cc:1397] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.003s
I20250625 13:52:23.069964 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "663588b7e9ee430faa180fec00897e22" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.070211 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_45daef3051ac480facff13fca3be7f75 [id=05bdba00acc94dfdaff1ecce7e0b3111]
I20250625 13:52:23.070549 983 leader_election.cc:304] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.070960 2883 raft_consensus.cc:3058] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.071041 1171 raft_consensus.cc:357] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.071219 1171 raft_consensus.cc:383] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.071287 1171 raft_consensus.cc:738] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.071251 1166 raft_consensus.cc:357] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.071552 1166 raft_consensus.cc:383] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.071553 1171 consensus_queue.cc:260] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.071664 1166 raft_consensus.cc:738] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.071996 1166 consensus_queue.cc:260] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.072911 1166 ts_tablet_manager.cc:1428] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:52:23.073032 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_6fadcfd061ab4fbdbd4bcbef13df8910 [id=06980d9c71814771ae6ccb685b4e54c2]
I20250625 13:52:23.073818 1166 tablet_bootstrap.cc:492] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.074501 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e3936edb9d8e47d08189e8903287c572 [id=06c61eaf6373492e8a72ba7d0442c601]
I20250625 13:52:23.074981 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_c9b44fa6c36f4b0fab24634ea2235568 [id=06e1a3dddf984b508ec13579bd9322b5]
I20250625 13:52:23.076270 2883 raft_consensus.cc:2747] T 663588b7e9ee430faa180fec00897e22 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.072228 1171 ts_tablet_manager.cc:1428] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 13:52:23.077576 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_afdcf859399e4102b6836c6ffe61ef4a [id=06fc6ee370e440c784261ae9944211e4]
I20250625 13:52:23.078481 1170 tablet_bootstrap.cc:492] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.078675 1170 ts_tablet_manager.cc:1397] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.002s
I20250625 13:52:23.079485 1170 raft_consensus.cc:357] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.079715 1170 raft_consensus.cc:383] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.079677 1166 tablet_bootstrap.cc:654] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.079883 1170 raft_consensus.cc:738] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.080168 1170 consensus_queue.cc:260] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.080610 1170 ts_tablet_manager.cc:1428] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:23.080942 1170 tablet_bootstrap.cc:492] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:23.082522 1170 tablet_bootstrap.cc:654] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.083838 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_847080232892457e98242244a03c75f6 [id=074a3f1be226435a90a11749bb29d5c3]
I20250625 13:52:23.084875 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_1ec3d8755b0d4d0f86175e3f35c24662 [id=079aa2bd25cd43f195377e4997fa5bc5]
I20250625 13:52:23.094244 1050 raft_consensus.cc:3058] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.096482 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3f19c3d36661484dac5b43b45c74851c [id=07bf9483068c420f8be59b01b93e5bbb]
I20250625 13:52:23.096982 1166 tablet_bootstrap.cc:492] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.097303 1166 ts_tablet_manager.cc:1397] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.001s
I20250625 13:52:23.097921 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_9c837b1557cb4c96a2016a326656a84f [id=080f781725ed4e45aa512416d769a961]
I20250625 13:52:23.098964 1166 raft_consensus.cc:357] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.099225 1166 raft_consensus.cc:383] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.099335 1166 raft_consensus.cc:738] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.099676 1166 consensus_queue.cc:260] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.100932 1166 ts_tablet_manager.cc:1428] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:23.101418 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_1bf18edd3d52428f9c11971f9df6b8a0 [id=08fa53445f5641e7af67692bc16764db]
I20250625 13:52:23.101959 1166 tablet_bootstrap.cc:492] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.103133 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_cd806d72367a416395dfb798a9ba8084 [id=095583ecc61145248d019934ecc6ed6e]
I20250625 13:52:23.104327 2659 raft_consensus.cc:491] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.104552 2659 raft_consensus.cc:513] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.104802 2775 consensus_queue.cc:1035] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.105679 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.105839 769 raft_consensus.cc:2466] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:23.106252 985 leader_election.cc:304] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:52:23.106990 2659 leader_election.cc:290] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.107815 1166 tablet_bootstrap.cc:654] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.119508 2883 raft_consensus.cc:2802] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 13:52:23.119838 2883 raft_consensus.cc:491] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election (detected failure of leader affa1c3035b940e88ca143be7d35840d)
I20250625 13:52:23.119959 2883 raft_consensus.cc:3058] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:52:23.121754 1171 tablet_bootstrap.cc:492] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:23.122655 2883 raft_consensus.cc:513] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.123442 1171 tablet_bootstrap.cc:654] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.123759 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.123934 769 raft_consensus.cc:3058] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:52:23.124408 2883 leader_election.cc:290] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 2 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.126613 769 raft_consensus.cc:2466] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 2.
I20250625 13:52:23.127038 1044 raft_consensus.cc:1238] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 2 FOLLOWER]: Rejecting Update request from peer affa1c3035b940e88ca143be7d35840d for earlier term 1. Current term is 2. Ops: []
I20250625 13:52:23.127617 985 leader_election.cc:304] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, 5c1aa07ab6bc4b0089e46951240a37c9; no voters:
I20250625 13:52:23.127743 2775 consensus_queue.cc:1046] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.128098 2883 raft_consensus.cc:2802] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:52:23.128099 2573 raft_consensus.cc:3053] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Stepping down as leader of term 1
I20250625 13:52:23.128166 2573 raft_consensus.cc:738] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.128284 2883 raft_consensus.cc:695] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [term 2 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:23.128446 2573 consensus_queue.cc:260] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.128634 2573 raft_consensus.cc:3058] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:52:23.128623 2883 consensus_queue.cc:237] T ccce48011a15405db2b95c04c9b01b3c P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.131289 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:23.131412 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccce48011a15405db2b95c04c9b01b3c" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.134613 914 raft_consensus.cc:2466] T ccce48011a15405db2b95c04c9b01b3c P affa1c3035b940e88ca143be7d35840d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 2.
I20250625 13:52:23.135608 769 raft_consensus.cc:1238] T ccce48011a15405db2b95c04c9b01b3c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 2 FOLLOWER]: Rejecting Update request from peer affa1c3035b940e88ca143be7d35840d for earlier term 1. Current term is 2. Ops: []
I20250625 13:52:23.149031 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_8e8fabf01ba94ac8b83358459a201156 [id=09be3b3816c641e9b0f5b7b1449c114b]
I20250625 13:52:23.149925 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_15e2aec48b1c4191a1ca7a3b48eac9ab [id=0a0b8545bca14836a8c5f102f328f9a3]
I20250625 13:52:23.150672 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d545aae2b146473aac4fcc9634d4a56a [id=0a89cc39e5f844249b8fa781dc5249b8]
I20250625 13:52:23.150682 2903 consensus_queue.cc:1035] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.151115 1166 tablet_bootstrap.cc:492] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.151222 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_b45de936fca240bd8d8d63cac7007f5e [id=0af3ac93e56643548b3da06116a13b43]
I20250625 13:52:23.151340 1166 ts_tablet_manager.cc:1397] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.008s
I20250625 13:52:23.152207 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_1b641ce25e2e4126bd10bcd3d7f8386c [id=0b47db3407dd439694c0bc3451a1b370]
I20250625 13:52:23.152994 1166 raft_consensus.cc:357] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.153252 1166 raft_consensus.cc:383] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.153342 1166 raft_consensus.cc:738] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.153668 1170 tablet_bootstrap.cc:492] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.153703 1166 consensus_queue.cc:260] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.153980 1170 ts_tablet_manager.cc:1397] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.001s
I20250625 13:52:23.155021 1166 ts_tablet_manager.cc:1428] T 17efe97d559a45dc9c0ca0b74efa7aa6 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 13:52:23.155493 1170 raft_consensus.cc:357] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.155824 1170 raft_consensus.cc:383] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.155961 1170 raft_consensus.cc:738] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.156330 1170 consensus_queue.cc:260] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.156930 1170 ts_tablet_manager.cc:1428] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:23.159129 1166 tablet_bootstrap.cc:492] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.162109 1170 tablet_bootstrap.cc:492] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:23.165218 1166 tablet_bootstrap.cc:654] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.168165 1170 tablet_bootstrap.cc:654] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.169464 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_1fb6c2cf80f242568459c13aae98c60f [id=0b60566c1ac44bf586c6208203056995]
I20250625 13:52:23.169950 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_7389669b9b714262b43171af80118da6 [id=0b6a5e62c8474868ba28d6229d947516]
I20250625 13:52:23.173552 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_19cabef12f194c91ab805ee4d346fbae [id=0b7eab7b5299492fa0eeebec89cf86fe]
I20250625 13:52:23.174110 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_6d7be80f24cc434c949137a402b4b7c7 [id=0c3cfe5b4048494f8cb814f29d799cd5]
I20250625 13:52:23.175774 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_0376b3d69a954c6881b4fd9d11e4b86b [id=0d141ebb7fc74195a5c05eeb73708142]
I20250625 13:52:23.176793 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ec46a4ca51884b13aa80806684369744 [id=0d9731930b4546c0b37dcac70722a0b8]
I20250625 13:52:23.182397 1166 tablet_bootstrap.cc:492] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.182736 1166 ts_tablet_manager.cc:1397] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.001s
I20250625 13:52:23.182973 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_065d55252ae74debb8216a63c480d278 [id=0e0850498f6b4137b90d78a16bb39fae]
I20250625 13:52:23.183457 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_29c1207ed89e4a33bceb77bc37580565 [id=0ea61479d0cc4670b609dd1d588b7cc1]
I20250625 13:52:23.184504 1166 raft_consensus.cc:357] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
W20250625 13:52:23.184999 983 consensus_peers.cc:487] T db6e9db2375144b38ec43ffca9ca4c1e P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Couldn't send request to peer affa1c3035b940e88ca143be7d35840d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db6e9db2375144b38ec43ffca9ca4c1e. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.185010 1166 raft_consensus.cc:383] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.185319 1166 raft_consensus.cc:738] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.185472 1170 tablet_bootstrap.cc:492] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.185617 1170 ts_tablet_manager.cc:1397] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250625 13:52:23.186043 1166 consensus_queue.cc:260] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.186403 1170 raft_consensus.cc:357] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.186591 1170 raft_consensus.cc:383] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.186658 1170 raft_consensus.cc:738] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.187793 1170 consensus_queue.cc:260] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.188282 1170 ts_tablet_manager.cc:1428] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 13:52:23.188567 1170 tablet_bootstrap.cc:492] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:23.190289 1170 tablet_bootstrap.cc:654] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.190867 1166 ts_tablet_manager.cc:1428] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250625 13:52:23.191216 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_f0e1bdf2998341a5988d7e7723151b3d [id=0eb88cded43b4b1fb0a745e1d66803cf]
I20250625 13:52:23.193007 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d568530b5b6e472fb3c3fd4b301a6dd2 [id=0f41315f3f4e46edb8f4b0f8a6b0f25a]
I20250625 13:52:23.200740 1166 tablet_bootstrap.cc:492] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.202844 2858 consensus_queue.cc:1035] T 48645c913fc24c03a922cf6ea7531022 P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.203603 1166 tablet_bootstrap.cc:654] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.208027 2775 consensus_queue.cc:1035] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.221799 2883 raft_consensus.cc:491] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.221968 2883 raft_consensus.cc:513] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.224066 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9efc7631df4ad29112a41f229ae2e9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.224306 915 raft_consensus.cc:2466] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 0.
W20250625 13:52:23.224848 985 leader_election.cc:343] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4c9efc7631df4ad29112a41f229ae2e9
I20250625 13:52:23.224319 2573 consensus_queue.cc:1035] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 13:52:23.226089 983 leader_election.cc:304] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:23.226449 2659 raft_consensus.cc:2802] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.226616 2883 leader_election.cc:290] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.226658 2659 raft_consensus.cc:491] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.226891 2659 raft_consensus.cc:3058] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.227936 1166 tablet_bootstrap.cc:492] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.228153 1166 ts_tablet_manager.cc:1397] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20250625 13:52:23.229835 1166 raft_consensus.cc:357] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.230245 1166 raft_consensus.cc:383] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.230379 1166 raft_consensus.cc:738] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.230811 1166 consensus_queue.cc:260] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.231779 2659 raft_consensus.cc:513] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.232247 1166 ts_tablet_manager.cc:1428] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250625 13:52:23.233173 1166 tablet_bootstrap.cc:492] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:23.233376 985 leader_election.cc:343] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Not found: Tablet not found: 4c9efc7631df4ad29112a41f229ae2e9
I20250625 13:52:23.234277 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9efc7631df4ad29112a41f229ae2e9" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:23.234504 915 raft_consensus.cc:3058] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.237897 915 raft_consensus.cc:2466] T 4c9efc7631df4ad29112a41f229ae2e9 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c008de91eb444f48da2c4f0b2d7a1ea in term 1.
I20250625 13:52:23.238433 983 leader_election.cc:304] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d; no voters: 5c1aa07ab6bc4b0089e46951240a37c9
I20250625 13:52:23.238556 1166 tablet_bootstrap.cc:654] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.238965 2883 raft_consensus.cc:2802] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.239136 2883 raft_consensus.cc:695] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 1 LEADER]: Becoming Leader. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Running, Role: LEADER
I20250625 13:52:23.239482 2883 consensus_queue.cc:237] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.241797 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_a04efddf105f4643ab4db1cd333eada5 [id=0f481a17699c4cbba748d4924dd13e06]
I20250625 13:52:23.243377 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_736e211328b54c2faf37ec980eae8d82 [id=0feadde1af1f419ea0e6e5523cb27b4b]
I20250625 13:52:23.243448 2858 raft_consensus.cc:491] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.243587 2858 raft_consensus.cc:513] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.244045 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_550cc88382f845bcad839c4bcd436274 [id=1022cdcee2c64f388a8abc8f6e48ce7a]
I20250625 13:52:23.244210 2858 leader_election.cc:290] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.245015 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb4973cbffc743dba98d7b7c9f76be3f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:23.245620 711 leader_election.cc:343] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.245942 2659 leader_election.cc:290] T 4c9efc7631df4ad29112a41f229ae2e9 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.245965 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb4973cbffc743dba98d7b7c9f76be3f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.246538 711 leader_election.cc:343] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.246654 711 leader_election.cc:304] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.246889 2858 raft_consensus.cc:2747] T fb4973cbffc743dba98d7b7c9f76be3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.257344 1166 tablet_bootstrap.cc:492] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.257709 1166 ts_tablet_manager.cc:1397] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.006s
I20250625 13:52:23.258975 1166 raft_consensus.cc:357] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.259361 1166 raft_consensus.cc:383] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.259536 1166 raft_consensus.cc:738] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.259984 1166 consensus_queue.cc:260] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.261032 1166 ts_tablet_manager.cc:1428] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:23.261967 1166 tablet_bootstrap.cc:492] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.266604 2573 consensus_queue.cc:1035] T 663588b7e9ee430faa180fec00897e22 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.267424 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_37db3d8c2a0d44e19c1598de8b337d6a [id=106eebd9055841afa03aa5f75c3b0e1d]
I20250625 13:52:23.268148 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ed63056903cc48609558b9b8419c6c8b [id=113def9728144b108b7485ade091ff9a]
I20250625 13:52:23.268635 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_46c504eaed8042dd930600225e3fe0b5 [id=116969aa3f7a4ef29b7007b14e1deddc]
I20250625 13:52:23.269236 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_18b980dca5124627abd9e0aa0476057d [id=1174afee7936469e80fcd17a3d5afeed]
I20250625 13:52:23.269999 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_50a6dcd0a6a948ceb2f7ffb547f0be00 [id=119dad3e0b334948b1bce47f1492cf59]
I20250625 13:52:23.270479 1166 tablet_bootstrap.cc:654] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.270967 1171 tablet_bootstrap.cc:492] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:23.271173 1171 ts_tablet_manager.cc:1397] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.150s user 0.004s sys 0.033s
I20250625 13:52:23.271986 1170 tablet_bootstrap.cc:492] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.272037 1171 raft_consensus.cc:357] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.272192 1170 ts_tablet_manager.cc:1397] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.002s
I20250625 13:52:23.272323 1171 raft_consensus.cc:383] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.272442 1171 raft_consensus.cc:738] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.272773 1171 consensus_queue.cc:260] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.272912 1170 raft_consensus.cc:357] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.273088 1170 raft_consensus.cc:383] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.273217 1170 raft_consensus.cc:738] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.273473 1170 consensus_queue.cc:260] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.274320 1170 ts_tablet_manager.cc:1428] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:23.273679 1171 ts_tablet_manager.cc:1428] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 13:52:23.274628 1170 tablet_bootstrap.cc:492] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:23.275513 1047 raft_consensus.cc:3058] T 3bc97a56aaf741648bae711cffacc6fb P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.278388 1166 tablet_bootstrap.cc:492] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.278590 2573 consensus_queue.cc:1035] T 3bc97a56aaf741648bae711cffacc6fb P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.278671 1166 ts_tablet_manager.cc:1397] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20250625 13:52:23.279342 1171 tablet_bootstrap.cc:492] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:23.279942 1166 raft_consensus.cc:357] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.280261 1166 raft_consensus.cc:383] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.280455 1166 raft_consensus.cc:738] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.280862 1166 consensus_queue.cc:260] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.281790 1166 ts_tablet_manager.cc:1428] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 13:52:23.282234 1166 tablet_bootstrap.cc:492] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.283419 1171 tablet_bootstrap.cc:654] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.285132 1166 tablet_bootstrap.cc:654] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.306039 1166 tablet_bootstrap.cc:492] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.306298 1166 ts_tablet_manager.cc:1397] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20250625 13:52:23.307134 1166 raft_consensus.cc:357] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.307482 1166 raft_consensus.cc:383] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.307615 1166 raft_consensus.cc:738] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.307917 1166 consensus_queue.cc:260] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.308624 1166 ts_tablet_manager.cc:1428] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 13:52:23.315673 1171 tablet_bootstrap.cc:492] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:23.315894 1171 ts_tablet_manager.cc:1397] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20250625 13:52:23.316716 1171 raft_consensus.cc:357] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.316978 1171 raft_consensus.cc:383] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.317080 1171 raft_consensus.cc:738] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.317602 1166 tablet_bootstrap.cc:492] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.318373 1171 consensus_queue.cc:260] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.319927 1171 ts_tablet_manager.cc:1428] T 29a542de77674e33b70647ffd9475fbf P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 13:52:23.320557 1166 tablet_bootstrap.cc:654] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.324698 1171 tablet_bootstrap.cc:492] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:23.325119 1050 raft_consensus.cc:3058] T 189a9347cd5d4b5f955e5c0706aa3947 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.327773 2573 raft_consensus.cc:491] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.328033 2573 raft_consensus.cc:513] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.329262 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:23.329878 849 leader_election.cc:343] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.330657 2573 leader_election.cc:290] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:23.331013 1166 tablet_bootstrap.cc:492] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.331287 1166 ts_tablet_manager.cc:1397] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20250625 13:52:23.331763 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d5c89472636b4d788a3a059b209f2361 [id=11fbd6ba0ae5490195d3326bf899cbd3]
I20250625 13:52:23.332348 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_42544de6446c4aed9bddadf23288264f [id=1239f72862034c4a8c5e12b2c8f112b3]
I20250625 13:52:23.330960 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.332621 1166 raft_consensus.cc:357] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.333009 1166 raft_consensus.cc:383] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.333278 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_0af46ccac2054224a7c0186bbc940d21 [id=12f3de56154643299cd8b3899711377e]
I20250625 13:52:23.333181 774 raft_consensus.cc:2466] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:23.333323 1166 raft_consensus.cc:738] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.333873 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_01f30a93171b4189adaa5b5c283a43e1 [id=13a2c478d08d4f3d80332697879d54d0]
I20250625 13:52:23.333984 851 leader_election.cc:304] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.334260 2573 raft_consensus.cc:2802] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.334344 2573 raft_consensus.cc:491] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.334424 2573 raft_consensus.cc:3058] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.334741 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_06ce6be66e684fce8aa585310d55aba2 [id=13a32c19e3f84808a5a3fa2f192b7984]
I20250625 13:52:23.334733 1166 consensus_queue.cc:260] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.335431 1171 tablet_bootstrap.cc:654] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.335415 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_7ae6d5c541cf45baaeb485b10d6c9f8a [id=13b265cdb0214da085594e59880edfc7]
I20250625 13:52:23.336503 1166 ts_tablet_manager.cc:1428] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 13:52:23.337563 1166 tablet_bootstrap.cc:492] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
I20250625 13:52:23.340996 2775 consensus_queue.cc:1035] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.341737 2573 raft_consensus.cc:513] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.342730 2573 leader_election.cc:290] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:23.343097 1170 tablet_bootstrap.cc:654] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.343472 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.343645 764 raft_consensus.cc:3058] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.344053 1166 tablet_bootstrap.cc:654] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.344875 2573 consensus_queue.cc:1035] T 189a9347cd5d4b5f955e5c0706aa3947 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.346319 764 raft_consensus.cc:2466] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.346976 851 leader_election.cc:304] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:23.347213 2775 raft_consensus.cc:2802] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.347373 2775 raft_consensus.cc:695] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.347657 2775 consensus_queue.cc:237] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.350137 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:23.350615 849 leader_election.cc:343] T 45d68de75ac84f1a8138c38fd51547cf P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.353919 2858 raft_consensus.cc:491] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.354074 2858 raft_consensus.cc:513] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.354650 2858 leader_election.cc:290] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.355156 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d89b0ddcc14148faa1a6926fcbb027a8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.355321 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d89b0ddcc14148faa1a6926fcbb027a8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.355854 711 leader_election.cc:343] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:23.356115 711 leader_election.cc:343] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.356230 711 leader_election.cc:304] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.356669 2858 raft_consensus.cc:2747] T d89b0ddcc14148faa1a6926fcbb027a8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.357463 2858 raft_consensus.cc:491] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.357620 2858 raft_consensus.cc:513] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.358196 2858 leader_election.cc:290] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.358808 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10350e0660f3406eaff0ea34b60c47f4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.359817 711 leader_election.cc:343] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.364905 2775 raft_consensus.cc:491] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.365032 2775 raft_consensus.cc:513] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.365952 2775 leader_election.cc:290] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:23.366189 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efc9536a716f4b9f9fc4af1637b3cd80" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.366258 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efc9536a716f4b9f9fc4af1637b3cd80" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.366362 774 raft_consensus.cc:2466] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:23.366662 851 leader_election.cc:304] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:23.366935 2775 raft_consensus.cc:2802] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.367023 2775 raft_consensus.cc:491] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250625 13:52:23.366948 849 leader_election.cc:343] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.367079 2775 raft_consensus.cc:3058] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.369701 2775 raft_consensus.cc:513] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
W20250625 13:52:23.380021 985 consensus_peers.cc:487] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69c44d72516e4883aff636ca65929c11. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.380924 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10350e0660f3406eaff0ea34b60c47f4" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.381572 2858 raft_consensus.cc:491] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.381779 2858 raft_consensus.cc:513] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.382555 2858 leader_election.cc:290] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.383430 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d25e4114a184bc99705042593ab4ab9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.384469 2659 raft_consensus.cc:491] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.384655 2659 raft_consensus.cc:513] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.384528 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d25e4114a184bc99705042593ab4ab9" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.385370 2659 leader_election.cc:290] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.386780 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd036ec08f774f5393067ce20bb07610" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.387636 711 leader_election.cc:343] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.387796 711 leader_election.cc:304] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
W20250625 13:52:23.388289 711 leader_election.cc:343] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.388381 2775 leader_election.cc:290] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
W20250625 13:52:23.388510 711 leader_election.cc:343] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.388633 711 leader_election.cc:304] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.388895 2903 raft_consensus.cc:2747] T 10350e0660f3406eaff0ea34b60c47f4 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.389078 2903 raft_consensus.cc:2747] T 4d25e4114a184bc99705042593ab4ab9 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.389128 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efc9536a716f4b9f9fc4af1637b3cd80" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:23.389608 849 leader_election.cc:343] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.390132 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efc9536a716f4b9f9fc4af1637b3cd80" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.390336 764 raft_consensus.cc:3058] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.396803 764 raft_consensus.cc:2466] T efc9536a716f4b9f9fc4af1637b3cd80 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.398216 851 leader_election.cc:304] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.398543 2775 raft_consensus.cc:2802] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.399751 2659 consensus_queue.cc:1035] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.400490 2775 raft_consensus.cc:695] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.400796 2775 consensus_queue.cc:237] T efc9536a716f4b9f9fc4af1637b3cd80 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.402307 1171 tablet_bootstrap.cc:492] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:23.402508 1171 ts_tablet_manager.cc:1397] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.004s
I20250625 13:52:23.403386 1171 raft_consensus.cc:357] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.403621 1171 raft_consensus.cc:383] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.403717 1171 raft_consensus.cc:738] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.403770 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3765a1c34b154d1f97898bad646c9ecf [id=1487a09c7ddd44bfb56c1cfbbf73be5c]
I20250625 13:52:23.403980 1171 consensus_queue.cc:260] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.405443 2659 consensus_queue.cc:1035] T 69c44d72516e4883aff636ca65929c11 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.405561 1171 ts_tablet_manager.cc:1428] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 13:52:23.405856 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd036ec08f774f5393067ce20bb07610" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
W20250625 13:52:23.406710 985 consensus_peers.cc:487] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 553af7c7326240838f37e3861d4e3357. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.406826 1171 tablet_bootstrap.cc:492] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:23.406953 985 leader_election.cc:304] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c008de91eb444f48da2c4f0b2d7a1ea; no voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.408259 2659 raft_consensus.cc:2747] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.414822 2858 raft_consensus.cc:491] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.415217 2858 raft_consensus.cc:513] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.416317 2858 leader_election.cc:290] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:23.416859 1171 tablet_bootstrap.cc:654] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.417227 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bf2f307ad7c4a6a856ddfb6d6fa2b57" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:23.417753 711 leader_election.cc:343] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.417375 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bf2f307ad7c4a6a856ddfb6d6fa2b57" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.419961 711 leader_election.cc:343] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.420174 711 leader_election.cc:304] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.420709 2858 raft_consensus.cc:2747] T 3bf2f307ad7c4a6a856ddfb6d6fa2b57 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.423949 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_81ed9530630f4ac9b888925f6f187d04 [id=15331889813d404d9f15fe761fd6642c]
I20250625 13:52:23.424687 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_8362565cb5f34b5fb4e5cb94da328683 [id=15bd5db613c34526be4805b87b190faf]
I20250625 13:52:23.425192 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e1a2fad7ba6143feb9d139ab60083e8a [id=169f630cf57647fe85bf3579a3c83667]
I20250625 13:52:23.425108 1171 tablet_bootstrap.cc:492] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:23.425509 1171 ts_tablet_manager.cc:1397] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250625 13:52:23.428467 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_9918069c31e945819ad208720537e6ea [id=16c3219eb49846b6b7450e5fdb3f7e07]
I20250625 13:52:23.429845 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_eb2c1659d4674c8c892d2a81049de520 [id=16e77db14c534086857f3c226330b65c]
I20250625 13:52:23.429596 1171 raft_consensus.cc:357] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.430403 1171 raft_consensus.cc:383] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.430845 1171 raft_consensus.cc:738] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
W20250625 13:52:23.431535 985 consensus_peers.cc:487] T 9be75d03121f4ea2906236ac3a5b0764 P 1c008de91eb444f48da2c4f0b2d7a1ea -> Peer 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801): Couldn't send request to peer 5c1aa07ab6bc4b0089e46951240a37c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9be75d03121f4ea2906236ac3a5b0764. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.431885 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_528b80e737c9462194cd9b50d91b7db9 [id=171d9555defd4d1ba8634fb0edfb55d3]
I20250625 13:52:23.431850 1171 consensus_queue.cc:260] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.432799 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_050b3df547a443f797e988c8fcb3dc97 [id=17cc7336c1144e4dbe7fe181b32646f5]
I20250625 13:52:23.434370 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_de9b2bd8421841fab7fff76f350bf461 [id=19abba73a8f34966afef3923120eda5e]
I20250625 13:52:23.434916 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d3f4f37b0b30453789b651b77c3b3833 [id=19cdf0d752584848af332cfdc0b491dd]
I20250625 13:52:23.435796 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_db68f5b8cfc64881a64265aa1240324e [id=1a1ec8e7044e414eb6154e505764fc3c]
I20250625 13:52:23.436334 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ae649101c4f34ca69ea84f0679906499 [id=1a88f159fd4b4af9a6ce5f1895ff43f5]
I20250625 13:52:23.437680 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_635e90a4272746cab8e8f9802215e784 [id=1ca6396d145c4b68a68e153cf78f8bcc]
I20250625 13:52:23.438230 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_57f80199b4354980b61485dafd3ebf7a [id=1ccdf11a8fe64810a774e47fb27d0916]
I20250625 13:52:23.438871 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ae346c0db6734c4684eb50013dda4b4a [id=1cd3f55ac4944dc2a43b80265ab9f7ac]
I20250625 13:52:23.438905 1171 ts_tablet_manager.cc:1428] T 6e8b9b732f8948d2b4c1a81a1de4489a P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250625 13:52:23.439729 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3d63596a7f60412eafc186aec5181d38 [id=1cea2560f77548b5b575d2cb0267c8e0]
I20250625 13:52:23.440254 1171 tablet_bootstrap.cc:492] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:23.441020 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_39f734a42d3845478cc2f4f5fe79181f [id=1d038a83b5764f99bc9dec7a4e444ac8]
I20250625 13:52:23.442340 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3038b7ccb4f54d2caecfb61667127b56 [id=1ed1f7751fcd492ca7a928c7a0f491c1]
I20250625 13:52:23.442822 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_12751a3c74f64b27b5baf3fe44befca1 [id=1f0e3ad33f3b470791d7c9a81477ae2f]
I20250625 13:52:23.443331 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_a6c0a9a6d69f4cb0aa50ddb77f00a141 [id=1f0fa3f954274c0e91ff95a6fc6dfa43]
I20250625 13:52:23.444819 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_717dce88b94246cab98733e4565a27b5 [id=214da52a59a148e3bee6ee3a40b85bb0]
I20250625 13:52:23.444972 2775 raft_consensus.cc:491] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.445120 2775 raft_consensus.cc:513] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.446422 2775 leader_election.cc:290] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:23.446732 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1e362b69fe40c0b6472b56f8cb9bcb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.446892 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1e362b69fe40c0b6472b56f8cb9bcb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.447175 764 raft_consensus.cc:2466] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
W20250625 13:52:23.447331 849 leader_election.cc:343] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.447932 851 leader_election.cc:304] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.448381 2775 raft_consensus.cc:2802] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.448504 2775 raft_consensus.cc:491] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.448629 2775 raft_consensus.cc:3058] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.448978 1171 tablet_bootstrap.cc:654] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.451584 2953 raft_consensus.cc:491] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3fec9d8a79054a5991ae90d8a3101e32)
I20250625 13:52:23.451818 2953 raft_consensus.cc:513] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fec9d8a79054a5991ae90d8a3101e32" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33015 } } peers { permanent_uuid: "58351267767f4d9a85f974ebc40db6fe" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45727 } } peers { permanent_uuid: "d85834f366fa42c294c9290af21ea111" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43307 } }
I20250625 13:52:23.452592 2953 leader_election.cc:290] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fec9d8a79054a5991ae90d8a3101e32 (127.0.104.62:33015), d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307)
I20250625 13:52:23.453720 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_25bef492a0bc4aafa4f94e4917f3ab79 [id=21be6f89b15f4cdea678f1e6c331026d]
I20250625 13:52:23.452682 2775 raft_consensus.cc:513] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.455686 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1e362b69fe40c0b6472b56f8cb9bcb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:23.456377 849 leader_election.cc:343] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.457535 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_c7c83caf0473417da3a8fc96412d92f0 [id=23aa99864007411ea7a81a5d5bdad4c6]
I20250625 13:52:23.458401 2775 leader_election.cc:290] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:23.462383 1050 raft_consensus.cc:3058] T cab4329d58404014833ae5029f8f296d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.466766 2775 consensus_queue.cc:1035] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.459383 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_8d0d72b50bfa43c6979e378ec08f2d12 [id=245bae57243f4930a52a93bce8ca48df]
I20250625 13:52:23.464159 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c1e362b69fe40c0b6472b56f8cb9bcb" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.468135 764 raft_consensus.cc:3058] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.471688 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_1ff8eb7b88c9486fb60f9ca445bc30d3 [id=25a7472e96b4448683b5fcf35e540048]
I20250625 13:52:23.472295 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d1c8466871174daea32ea7d8bac86daa [id=25ae672e5ac043f2a1b5ca08437d325b]
I20250625 13:52:23.472869 764 raft_consensus.cc:2466] T 2c1e362b69fe40c0b6472b56f8cb9bcb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.473877 851 leader_election.cc:304] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.474354 2580 raft_consensus.cc:2802] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.474507 2580 raft_consensus.cc:695] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.474784 2580 consensus_queue.cc:237] T 2c1e362b69fe40c0b6472b56f8cb9bcb P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.477754 2774 consensus_queue.cc:1035] T 535812d46d6444fda85d9767c6ca7e62 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.482059 1170 tablet_bootstrap.cc:492] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.482218 1170 ts_tablet_manager.cc:1397] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.208s user 0.004s sys 0.004s
I20250625 13:52:23.482995 1170 raft_consensus.cc:357] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.483196 1170 raft_consensus.cc:383] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.483263 1170 raft_consensus.cc:738] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.483507 1170 consensus_queue.cc:260] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.484153 1170 ts_tablet_manager.cc:1428] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 13:52:23.484522 1170 tablet_bootstrap.cc:492] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:23.485102 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_32b73fb2c3b84d13b81857f6d4a460e5 [id=26b2b91707a344ec987d4a9f3b0f0790]
I20250625 13:52:23.487604 2858 raft_consensus.cc:491] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.487785 2858 raft_consensus.cc:513] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.488428 2858 leader_election.cc:290] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.489930 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25c71f11222e48818ea94984a985d6cb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:23.490710 711 leader_election.cc:343] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.491366 1170 tablet_bootstrap.cc:654] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.500831 2858 raft_consensus.cc:491] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.501085 2858 raft_consensus.cc:513] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.504144 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_6b02faeef1a54d03b3fedf55f1bbbc8a [id=277abdfbb8fe4ab4a8cea23b2f0849f5]
I20250625 13:52:23.506546 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d4ecf658368d43e2ba8c6e755e51b4e3 [id=27bb5b5377e2489c9fda8d0d776ef41e]
W20250625 13:52:23.508445 513 leader_election.cc:336] T 00000000000000000000000000000000 P 58351267767f4d9a85f974ebc40db6fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d85834f366fa42c294c9290af21ea111 (127.0.104.60:43307): Network error: Client connection negotiation failed: client connection to 127.0.104.60:43307: connect: Connection refused (error 111)
I20250625 13:52:23.510210 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e90bcf7a778340f286355f44c53e903c [id=2813ca06dbcf4f96bf0a93680fcb0c9f]
I20250625 13:52:23.513048 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_03fee4b89d0c4186beb0de611b2fc05f [id=286bb158e3de4eb79bc1f6c9bd117ae4]
I20250625 13:52:23.514849 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_8cae88538eb141a281ef6a3405a519aa [id=28a995512a2a4ba2aa8e243335f4d6e2]
I20250625 13:52:23.516239 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ee7a863cc44d4067a1f5a29e6fe267af [id=28eab56011a1491abfc13e26d286d5d7]
I20250625 13:52:23.517885 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_efee3de551ce4c8c96c15ea79ad7fc77 [id=28ff914e41654516a1db0b21d5e28124]
I20250625 13:52:23.519197 2858 leader_election.cc:290] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.519280 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_aef664c19f8a47569ac63d700d711236 [id=29646aaff9444c0ca835b4722d08875e]
I20250625 13:52:23.519943 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "150af49427ae47b98e9b2077bbbe1e2b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.520915 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_bdc29ce3c8744673b2c3c09c5f27acd5 [id=2998facb1b8d419c81fcbaad537c7bed]
W20250625 13:52:23.521462 711 leader_election.cc:343] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.522029 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "150af49427ae47b98e9b2077bbbe1e2b" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.522637 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_262ec080b91d44d68afd7d9c6341cb25 [id=29c869a5bbe04125ac60195cb120d423]
W20250625 13:52:23.523568 711 leader_election.cc:343] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.523756 711 leader_election.cc:304] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.524209 2858 raft_consensus.cc:2747] T 150af49427ae47b98e9b2077bbbe1e2b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.524542 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_70e8ca6bcdff497daaf2b4dcc9a8183e [id=29c8dc9e4e504fe282078307b9a4a2c0]
I20250625 13:52:23.526590 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_c183e5f797384e9993de44fd16071209 [id=29d11ad122834f9fa197f1ff2f27b3d4]
I20250625 13:52:23.529740 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_20830d90ca194d2d8a472b90228ae5b0 [id=2b83edcabfbb479d936e8802a569beb8]
I20250625 13:52:23.531201 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_957dc0554e3040d38f97ccfa311f4270 [id=2b85ef988fc74560bd3080a5e5bde122]
I20250625 13:52:23.532670 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_fdfdf918b84c4bc7874ae0a62279c7c7 [id=2bf141a440794be1919feb01f47d5009]
I20250625 13:52:23.533706 2858 raft_consensus.cc:491] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.533960 2858 raft_consensus.cc:513] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.534795 2858 leader_election.cc:290] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.535557 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f49c0fc97d44b28acfd6cb493906edb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.535842 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d5806194428c4cb9927a09061a2d6533 [id=2c3f0413818e41358b7952e366951cb9]
I20250625 13:52:23.536473 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f49c0fc97d44b28acfd6cb493906edb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.537109 711 leader_election.cc:343] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:23.537387 711 leader_election.cc:343] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.537537 711 leader_election.cc:304] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.539043 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_5c84ad3ea67d4b2cbdc4fb4eb0ea2c32 [id=2c706b062d2c4eb7ad30242b4b177a1f]
I20250625 13:52:23.540616 2858 raft_consensus.cc:2747] T 5f49c0fc97d44b28acfd6cb493906edb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.541067 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e99c95246c7443afa66ac138a1bf6f9a [id=2c9eaa20a3ed4ec1a63e7d425d1d94e6]
I20250625 13:52:23.542636 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_6dc8600ee239471dbe4e422ce34db986 [id=2cb3ccc7134642fe9747bd432d8609d7]
I20250625 13:52:23.544130 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_4ecc6555bb17447bb819b95c6b928c7b [id=2d6c8e320f1243b88a9d1a954f5d4804]
I20250625 13:52:23.546458 1170 tablet_bootstrap.cc:492] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.546659 1170 ts_tablet_manager.cc:1397] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.001s
I20250625 13:52:23.547260 2775 raft_consensus.cc:491] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.547461 2775 raft_consensus.cc:513] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.547508 1170 raft_consensus.cc:357] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.548265 2838 raft_consensus.cc:491] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.548384 2838 raft_consensus.cc:513] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.548755 2775 leader_election.cc:290] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:23.548837 2838 leader_election.cc:290] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:23.549573 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35aedb3b590d4917bac8fe0962d68fb3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.549741 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e73bea55af9e4890bf7070b488313c3f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.549772 769 raft_consensus.cc:2466] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:23.548530 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e73bea55af9e4890bf7070b488313c3f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.549517 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35aedb3b590d4917bac8fe0962d68fb3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.550042 774 raft_consensus.cc:2466] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:23.550200 851 leader_election.cc:304] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
W20250625 13:52:23.550374 849 leader_election.cc:343] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.550403 1170 raft_consensus.cc:383] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 13:52:23.550572 849 leader_election.cc:343] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.550583 1170 raft_consensus.cc:738] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.550745 2580 raft_consensus.cc:2802] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.550818 2580 raft_consensus.cc:491] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.550873 2580 raft_consensus.cc:3058] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.550911 1170 consensus_queue.cc:260] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.553705 2580 raft_consensus.cc:513] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.554229 2580 leader_election.cc:290] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291)
I20250625 13:52:23.554908 851 leader_election.cc:304] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.555112 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35aedb3b590d4917bac8fe0962d68fb3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:23.555373 2838 raft_consensus.cc:2802] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.555423 1170 ts_tablet_manager.cc:1428] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 13:52:23.555452 2838 raft_consensus.cc:491] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.555509 2838 raft_consensus.cc:3058] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:23.555732 849 leader_election.cc:343] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.558060 2838 raft_consensus.cc:513] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.558581 2838 leader_election.cc:290] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:23.558717 2838 consensus_queue.cc:1035] T cab4329d58404014833ae5029f8f296d P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.560207 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e73bea55af9e4890bf7070b488313c3f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:23.561478 849 leader_election.cc:343] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.563710 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3cb5cc5957974c9580239ecdf22f26cf [id=2e7bb5d0efcc4a17a42fe9c8435025ce]
I20250625 13:52:23.561133 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e73bea55af9e4890bf7070b488313c3f" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.564560 774 raft_consensus.cc:3058] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.565167 1170 tablet_bootstrap.cc:492] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:23.566511 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3348879866c946209d1b8e1d87153e2f [id=2ee8b46428b94860b6c81a59aeb00203]
I20250625 13:52:23.567060 1170 tablet_bootstrap.cc:654] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.569767 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e6f0bb3d8be34d36a5c48eeb803e5053 [id=2f3486b484e141fbad6554982346aea8]
I20250625 13:52:23.574301 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_01b8d89dbb714bdc83d42185cc4a1fd3 [id=2f413e1c59c147868eecda4e62a2232a]
I20250625 13:52:23.576401 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_30f7d431b5314e83b313365e3b0bffd5 [id=2f79cff5bb3e4be99f16ae477d3c6685]
I20250625 13:52:23.579123 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_60d3522fb58e4d2d8478eec6df099e91 [id=2f7c46bb4abc45d4b800b144cdd4874f]
I20250625 13:52:23.582793 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_03beb73b975f4565b8cecb57cc54e14d [id=2fe7121baaa140ca8670b9e39e572fea]
I20250625 13:52:23.584648 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_7eb6a8becb2843348ce594ffa2ee291c [id=30dcfcb4791149ce9f606d636876227e]
I20250625 13:52:23.586575 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_36b5354e955b4364845c9fbb1e93d0eb [id=3162c98fa7b3456e857237ae5f01756a]
I20250625 13:52:23.588276 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d9bf47b9c43c456e997fd5950c43f132 [id=316e544ee11a45ea9ec3d6bc804e356b]
I20250625 13:52:23.591390 774 raft_consensus.cc:2466] T e73bea55af9e4890bf7070b488313c3f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.591184 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_bb6eed6d946d4e7097bf4665b269c72a [id=325dd21568534ee58900ef0b8d79fae7]
I20250625 13:52:23.592674 1050 raft_consensus.cc:3058] T bd036ec08f774f5393067ce20bb07610 P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.594909 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d7faba35e83944d586e1e53d4b524950 [id=33400d976688476eb4450525402f12be]
I20250625 13:52:23.597394 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_fe7c9a0a684b4277a14d8ed100f3fa74 [id=33bfa70f7d944ff1b7e808288266b1f8]
I20250625 13:52:23.600059 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_8700871caf544559b569345d27b54935 [id=3469c1060d6e428ea47470b00e8ea00d]
I20250625 13:52:23.601465 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_491bf1ae35e74854b2d7f3f716215bbb [id=34c3e009fdfd4b5b88a57d640313b0d3]
I20250625 13:52:23.603049 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_9f776c4576a64684b3f3387da0b9923b [id=34ca94f083ca404bbd38efafeca91d6f]
I20250625 13:52:23.605610 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_8a3bb463510942988dfd2cb114c96e48 [id=359a935f26f8424e94c368599b410622]
I20250625 13:52:23.608553 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_f471ebec4d67405899f155aad9d3ec83 [id=35ca80beadf742df969e352ff737586d]
I20250625 13:52:23.612219 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_899784ce715e4e229d6d804c9a94cbcd [id=36477ab15a554593906071142e6cd546]
I20250625 13:52:23.613544 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_7c5253f2510845c58dc974a5a0840b62 [id=37fda201e6e54220873b8631039f6765]
I20250625 13:52:23.613838 2638 raft_consensus.cc:491] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.614017 2638 raft_consensus.cc:513] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.614692 2638 leader_election.cc:290] T 45d68de75ac84f1a8138c38fd51547cf P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.615356 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_58e294098e864cd5a9dd39dcbf23b881 [id=3903e757b0fb44438a1d0ad162934185]
I20250625 13:52:23.616073 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_69c35651e2b2452894ec73fae8eea1bd [id=3954de3444ba494388130f23dbdad5cc]
I20250625 13:52:23.616707 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_b0e2772ddddc4faf81571778a3405e88 [id=396b52a757ec48d4b185c395712081ab]
I20250625 13:52:23.617688 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_727a4473adab487fa65f094237ec794b [id=3977ee3968d246b7a8aed1fafd1822b7]
I20250625 13:52:23.618387 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_7531d8f57e9e4b4593b566e0362aa281 [id=39a6012eddee493a8ee508f20f320398]
I20250625 13:52:23.619498 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_7a3b5892cacf49f29167f2125d14f34a [id=3a329f34821443e6beb7b59680575dcb]
I20250625 13:52:23.620607 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3cf16642852d4afa96c4425698093888 [id=3a65e259b60046d094c1ab12973ecffd]
I20250625 13:52:23.622040 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_29f4b702a71c42d38dc12caed1b9e4b0 [id=3b2db2023698426486defcd332d7b109]
I20250625 13:52:23.624553 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.625685 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e46d21f14cf14f19b7944197845d51ea [id=3b6f30942ccf4b3589758fb1463b0f21]
I20250625 13:52:23.626359 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_7e3cd23ead3f4afa889078106d9b4711 [id=3bfbc121faf44d6691a73639838194c1]
I20250625 13:52:23.627614 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_9288bcd2d4bc4fdf90ea3fc0f27be760 [id=3c6db0d6e3fb4314ba1dddf8654c2386]
I20250625 13:52:23.628372 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_2009356d754d48c9a01a3058fe1ac4c5 [id=3d206a6b57d34bf599390f47c855c11a]
I20250625 13:52:23.629014 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_da70d46ae8854be99605668d2d0fdf95 [id=3d42f18b1b0049ed9011fdd3f7ec2724]
I20250625 13:52:23.629658 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_8a9e2586f6d0486faafe1b8311f52962 [id=3eaa6f5e5a5345f08748147d1880728b]
I20250625 13:52:23.630662 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d6a073fafd8a4d70951c67d16e010f29 [id=3ee1e37050ad4e958797be261e5dc440]
I20250625 13:52:23.632087 2638 consensus_queue.cc:1035] T 553af7c7326240838f37e3861d4e3357 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.635922 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_2a8a7faddcd5494fa349ad6ecc6aec18 [id=3ee8fb3325074059aeed265c750e3d94]
I20250625 13:52:23.636590 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ff092aaea4a944aba8072d10f3b7ce49 [id=3f116c6f6ae94239941fa04a335853ef]
I20250625 13:52:23.637522 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_6a309ccf2fff4b8fae82108cf1b95886 [id=4055346986414eec9af02674c2dc1628]
I20250625 13:52:23.637840 2638 consensus_queue.cc:1035] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.639261 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_c3e15d16f1e74232b39b0c990c64bb6e [id=4139a2f9cfc84892b42c875c6fd66d6e]
I20250625 13:52:23.640281 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ce4451ff102a4c22ba9539daffd65aa6 [id=4148344340d545ae8f071f8292500291]
I20250625 13:52:23.642468 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_107ccce5d138433db4fb8351f69e40a0 [id=423c4d58d233464ebe4fcd7a7d426f64]
I20250625 13:52:23.644137 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_efc036d6fcdd445c9420a616e7ee8469 [id=425d7c5e4b0b4fb69c3b61fc1ca233fe]
I20250625 13:52:23.645352 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_45c7b1313316471f9fc8a44c2f19554b [id=42a1c7c209384e59aa37028ffbcd2a41]
I20250625 13:52:23.647343 2775 consensus_queue.cc:1035] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.652230 2573 raft_consensus.cc:491] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.652350 2573 raft_consensus.cc:513] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.652899 2573 leader_election.cc:290] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:23.653781 849 consensus_peers.cc:487] T 0012e95ff4924723a6e6aaf6b85da438 P affa1c3035b940e88ca143be7d35840d -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.654042 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25c71f11222e48818ea94984a985d6cb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.654210 916 raft_consensus.cc:2466] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 0.
I20250625 13:52:23.654349 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b3d5a2ab7042c2a2f18890363e7a9b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:23.654948 849 leader_election.cc:343] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.655395 2638 consensus_queue.cc:1035] T 7efc8df3c8e24020b073a1b61bc9e0a0 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.615527 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d68de75ac84f1a8138c38fd51547cf" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.657943 778 raft_consensus.cc:2391] T 45d68de75ac84f1a8138c38fd51547cf P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c008de91eb444f48da2c4f0b2d7a1ea in current term 1: Already voted for candidate affa1c3035b940e88ca143be7d35840d in this term.
I20250625 13:52:23.618330 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35aedb3b590d4917bac8fe0962d68fb3" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.647104 2858 consensus_queue.cc:1035] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.654641 711 leader_election.cc:304] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.659229 2858 raft_consensus.cc:2802] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.659360 2858 raft_consensus.cc:491] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.659475 2858 raft_consensus.cc:3058] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.661026 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b3d5a2ab7042c2a2f18890363e7a9b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.661222 769 raft_consensus.cc:2466] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:23.661957 851 leader_election.cc:304] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.661979 2858 raft_consensus.cc:513] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.662495 2858 leader_election.cc:290] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.662499 2774 raft_consensus.cc:2802] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.662583 2774 raft_consensus.cc:491] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.662658 2774 raft_consensus.cc:3058] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.663223 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25c71f11222e48818ea94984a985d6cb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"
I20250625 13:52:23.663398 913 raft_consensus.cc:3058] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.666046 913 raft_consensus.cc:2466] T 25c71f11222e48818ea94984a985d6cb P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1aa07ab6bc4b0089e46951240a37c9 in term 1.
I20250625 13:52:23.666208 2774 raft_consensus.cc:513] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.666697 711 leader_election.cc:304] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:23.667078 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25c71f11222e48818ea94984a985d6cb" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:23.667465 764 raft_consensus.cc:3058] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.667634 2774 leader_election.cc:290] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:23.667840 711 leader_election.cc:343] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.667809 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b3d5a2ab7042c2a2f18890363e7a9b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
W20250625 13:52:23.668349 849 leader_election.cc:343] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.668488 2903 raft_consensus.cc:2802] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.668597 2903 raft_consensus.cc:695] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Running, Role: LEADER
I20250625 13:52:23.668843 2903 consensus_queue.cc:237] T 25c71f11222e48818ea94984a985d6cb P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.669229 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b3d5a2ab7042c2a2f18890363e7a9b" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.669410 769 raft_consensus.cc:3058] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
W20250625 13:52:23.672822 711 consensus_peers.cc:487] T 17b172c7631048ceb97ccbea896f7f5b P 5c1aa07ab6bc4b0089e46951240a37c9 -> Peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Couldn't send request to peer 1c008de91eb444f48da2c4f0b2d7a1ea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 13:52:23.675362 851 leader_election.cc:304] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.679133 2774 raft_consensus.cc:2802] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.680629 2774 raft_consensus.cc:695] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.681277 2774 consensus_queue.cc:237] T e73bea55af9e4890bf7070b488313c3f P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.687737 769 raft_consensus.cc:2466] T e3b3d5a2ab7042c2a2f18890363e7a9b P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.688485 851 leader_election.cc:304] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.689015 2573 raft_consensus.cc:2802] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.689149 2573 raft_consensus.cc:695] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.689416 2573 consensus_queue.cc:237] T e3b3d5a2ab7042c2a2f18890363e7a9b P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.699455 2883 consensus_queue.cc:1035] T e7bd4939714a4359846345ed0521701c P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.707036 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_6d9870087cdc4dfca8fa438020887d80 [id=43487ee990e640bcbeefc7c142075998]
I20250625 13:52:23.710340 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_05a86981b085438083683deb71ee651d [id=4394b6fcfdf943a9928991a8f2825126]
I20250625 13:52:23.718132 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_f7f62312f5ed44fa821485933ffc5d0b [id=439c4d4ad58c40faad7669c9c58218af]
I20250625 13:52:23.720291 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_c0213ee08c5b4b58bc22aec0259011e9 [id=43af99dd61534e2aaba6c9f442ca34a4]
I20250625 13:52:23.721983 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3a36675ae59b4414bee46d495834a281 [id=4461a624c8ac4ac8b82ae0c570a551ac]
I20250625 13:52:23.722669 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3ddbb86e81474e9ca79571a9c0372b74 [id=449814f10d87435e9e60bf524bc61f9e]
I20250625 13:52:23.731209 1170 tablet_bootstrap.cc:492] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea: No bootstrap required, opened a new log
I20250625 13:52:23.731474 1170 ts_tablet_manager.cc:1397] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent bootstrapping tablet: real 0.166s user 0.012s sys 0.026s
I20250625 13:52:23.731771 764 raft_consensus.cc:2466] T 35aedb3b590d4917bac8fe0962d68fb3 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.732973 1170 raft_consensus.cc:357] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.733132 851 leader_election.cc:304] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.733318 1170 raft_consensus.cc:383] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.733433 2573 raft_consensus.cc:2802] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.733436 1170 raft_consensus.cc:738] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c008de91eb444f48da2c4f0b2d7a1ea, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.733554 2573 raft_consensus.cc:695] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
I20250625 13:52:23.733774 2573 consensus_queue.cc:237] T 35aedb3b590d4917bac8fe0962d68fb3 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } } peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } }
I20250625 13:52:23.733757 1170 consensus_queue.cc:260] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.734054 2903 consensus_queue.cc:1035] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.736884 1171 tablet_bootstrap.cc:492] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:23.737126 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_3fd8fa584be64d3cbd13d977cc8f9719 [id=453abb550d3d4d6694a3eeae15012900]
I20250625 13:52:23.737417 1171 ts_tablet_manager.cc:1397] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.298s user 0.022s sys 0.058s
I20250625 13:52:23.739465 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_34cd80c2997d41c281b3eb2b2c5420c9 [id=460e1c8a50464b3f876b89d48a182797]
I20250625 13:52:23.739984 1171 raft_consensus.cc:357] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.740449 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_c4f1a08649504a7db750c88b65ff985c [id=4631cf687a9c478ba42e9ea3375ca639]
I20250625 13:52:23.740624 1171 raft_consensus.cc:383] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.740826 1171 raft_consensus.cc:738] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.741468 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ddde325144324d1f88fd7d68629a7323 [id=4668bfd2e89b4179b1503e31c6c8276a]
I20250625 13:52:23.741618 1171 consensus_queue.cc:260] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.742734 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_78e65c78bd4d43f0900060ea0aa8203a [id=479b215e18984958bdd49fe60c2fc6ab]
I20250625 13:52:23.743952 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_35471795fa7d4688ae966d55ca175371 [id=47c01d70221c45bcabe0185a84119ae4]
I20250625 13:52:23.743906 1171 ts_tablet_manager.cc:1428] T 150af49427ae47b98e9b2077bbbe1e2b P affa1c3035b940e88ca143be7d35840d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 13:52:23.745360 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_9998378b240c46bcb121279c983960cc [id=48466b87e16942b59e1a05c811ecfd5a]
I20250625 13:52:23.745808 1171 tablet_bootstrap.cc:492] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d: Bootstrap starting.
I20250625 13:52:23.746980 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_d6cb85bab6c44d7bb80f746165017e62 [id=48951dc7f41341218c139e03c57416de]
I20250625 13:52:23.747826 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ae093472c1b64d16bdfd6fd142faca11 [id=48a8063c68914a71bcc4fcdb97d7a0e6]
I20250625 13:52:23.747730 1166 tablet_bootstrap.cc:492] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9: No bootstrap required, opened a new log
I20250625 13:52:23.748714 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e3e6c0c72937408cb2125f54908cbb0e [id=48ae56f42fe14313ba05232f01f825bf]
I20250625 13:52:23.748917 1166 ts_tablet_manager.cc:1397] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent bootstrapping tablet: real 0.412s user 0.029s sys 0.120s
I20250625 13:52:23.749593 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_231ba9d7a8614e20b45715e4a09b9437 [id=48ba1774b6ae43c49493d4547f1d8650]
I20250625 13:52:23.750523 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_81625f4255fe4302808c19d2c855b6e3 [id=4971f9f659ab4f9ab9332056e8f4055a]
I20250625 13:52:23.752152 2903 raft_consensus.cc:491] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.752336 2903 raft_consensus.cc:513] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.753175 2903 leader_election.cc:290] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.754549 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4be0bc23cde4f0f94bd678096e0c7d8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
W20250625 13:52:23.755363 711 leader_election.cc:343] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.756083 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_f9b2ff84e19a476bb879930b39eaa142 [id=49c81f046b034d91873a1051786c1c32]
I20250625 13:52:23.756968 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_0b55a0fa335e4435a1f8c85d61d7c13b [id=4a377b59e94f408c871c5a1839d18e83]
I20250625 13:52:23.755632 1166 raft_consensus.cc:357] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.757923 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_56e5c59a581d431cada560e8606965ba [id=4aa0d48db54c44b68d36dd440cc11119]
I20250625 13:52:23.757640 1166 raft_consensus.cc:383] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.758776 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_612bc80f50694afebddc36a5f827e377 [id=4aac9020498342f2bc1062c64f01f716]
I20250625 13:52:23.760172 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_ac4439707ad3409595f54dce3c9b810f [id=4b7fe0ca3a3044a0b82fd38b4bfa191a]
I20250625 13:52:23.761029 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_52de821c06b24eea87c7839e7417a783 [id=4b8fad3e0cfd4e0a8d68946deb48a28f]
I20250625 13:52:23.762986 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_190dea7daa454753b901b65cd213f592 [id=4c0aa0b19c18488f8a3ba2b2dc9a688a]
I20250625 13:52:23.763921 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_a4a8e4337fea4493bd07b556314e49db [id=4ce18d456651499281dca929c9c8279b]
I20250625 13:52:23.765551 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_b32fea3ffed64728b8b1b4a86b44c628 [id=4d9dc7e6615940e38f7c61644a03022a]
I20250625 13:52:23.767576 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_437ef29da8a64a32a591093ac536eadc [id=4f1d1e14e48c4849a326f2afe02d2119]
I20250625 13:52:23.768522 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_aeb56ffe1b1343528965b690ef571cc7 [id=4f23bc58143640329474c296ba27e8c5]
I20250625 13:52:23.768898 2659 raft_consensus.cc:491] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.769074 2659 raft_consensus.cc:513] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.769680 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_4ef4bb15e00b423ba18232d0feb46e36 [id=4f2def4377f841b0be3657b4ee82f73d]
I20250625 13:52:23.769794 2659 leader_election.cc:290] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.779693 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf7e30cdff245208dacc6963fe88c7d" candidate_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.781682 1171 tablet_bootstrap.cc:654] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.783490 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4be0bc23cde4f0f94bd678096e0c7d8" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.784137 1044 raft_consensus.cc:3058] T cbf7e30cdff245208dacc6963fe88c7d P 1c008de91eb444f48da2c4f0b2d7a1ea [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.786947 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_e43ad6762bd943779162fd6eddfe14f4 [id=507e2802010849108aa302ab923228c7]
I20250625 13:52:23.790899 2775 raft_consensus.cc:491] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.791029 2775 raft_consensus.cc:513] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.758877 1166 raft_consensus.cc:738] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1aa07ab6bc4b0089e46951240a37c9, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.762118 2903 raft_consensus.cc:491] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.791438 2903 raft_consensus.cc:513] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.791601 2775 leader_election.cc:290] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
I20250625 13:52:23.792032 2903 leader_election.cc:290] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.774901 2693 raft_consensus.cc:491] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:52:23.792213 2693 raft_consensus.cc:513] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.792330 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6d0dbbe2894e61b0ea08f98bce61e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" is_pre_election: true
I20250625 13:52:23.792511 764 raft_consensus.cc:2466] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 0.
I20250625 13:52:23.792755 2693 leader_election.cc:290] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247)
I20250625 13:52:23.792711 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6d0dbbe2894e61b0ea08f98bce61e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
W20250625 13:52:23.787642 711 leader_election.cc:343] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.792886 711 leader_election.cc:304] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.793162 851 leader_election.cc:304] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters:
I20250625 13:52:23.793260 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_eaa443a9124c4346a760363614d32112 [id=509889c267d148ebb93b1f74c03ce336]
I20250625 13:52:23.793404 2775 raft_consensus.cc:2802] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:52:23.793494 2775 raft_consensus.cc:491] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:52:23.793555 2775 raft_consensus.cc:3058] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.793769 2693 raft_consensus.cc:2747] T b4be0bc23cde4f0f94bd678096e0c7d8 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.793954 2693 consensus_queue.cc:1035] T cbf7e30cdff245208dacc6963fe88c7d P 5c1aa07ab6bc4b0089e46951240a37c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:52:23.794698 1166 consensus_queue.cc:260] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.794647 2994 consensus_queue.cc:1035] T af704cdf44ef42f7b5ea10b199d0f017 P 1c008de91eb444f48da2c4f0b2d7a1ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 13:52:23.795380 1166 ts_tablet_manager.cc:1428] T d85cdadfd694486590092a7b6f61f44c P 5c1aa07ab6bc4b0089e46951240a37c9: Time spent starting tablet: real 0.046s user 0.003s sys 0.000s
I20250625 13:52:23.796054 1166 tablet_bootstrap.cc:492] T dc1c9cea3d5a405a958c6c7585dbef39 P 5c1aa07ab6bc4b0089e46951240a37c9: Bootstrap starting.
W20250625 13:52:23.797000 849 leader_election.cc:343] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.797717 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa16fa387b2a4bb5899b1487642c48d0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.798177 1166 tablet_bootstrap.cc:654] T dc1c9cea3d5a405a958c6c7585dbef39 P 5c1aa07ab6bc4b0089e46951240a37c9: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.798368 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34371795c8094e73ab4e5dbeedb7aa2f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" is_pre_election: true
I20250625 13:52:23.798710 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_843c4743b1b346f9a3fa1343436191ac [id=5113ee3dc80346839a305b39789caad9]
W20250625 13:52:23.802713 711 leader_election.cc:343] T fa16fa387b2a4bb5899b1487642c48d0 P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
W20250625 13:52:23.803501 711 leader_election.cc:343] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.805850 1170 ts_tablet_manager.cc:1428] T 1310fee26fe642bf9919a2bd8e1c091f P 1c008de91eb444f48da2c4f0b2d7a1ea: Time spent starting tablet: real 0.074s user 0.002s sys 0.002s
I20250625 13:52:23.806221 1170 tablet_bootstrap.cc:492] T 0012e95ff4924723a6e6aaf6b85da438 P 1c008de91eb444f48da2c4f0b2d7a1ea: Bootstrap starting.
I20250625 13:52:23.807976 1170 tablet_bootstrap.cc:654] T 0012e95ff4924723a6e6aaf6b85da438 P 1c008de91eb444f48da2c4f0b2d7a1ea: Neither blocks nor log segments found. Creating new log.
I20250625 13:52:23.808241 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_46e5b7c336504c6a87427975a0168a63 [id=515860fa91ae43bea4530116b0306433]
I20250625 13:52:23.811337 1171 tablet_bootstrap.cc:492] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d: No bootstrap required, opened a new log
I20250625 13:52:23.811600 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_eb12aa64a1cb41b4a47f82c64e510a70 [id=5170771f5f5f41fd8eca72c7899c417b]
I20250625 13:52:23.811923 2775 raft_consensus.cc:513] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.817188 2747 catalog_manager.cc:671] Loaded metadata for table default.Table_001eda8fd8bf4b5b884fc36818b52fe0 [id=52235c623d184d4ebc54089f44d4a022]
I20250625 13:52:23.812222 1171 ts_tablet_manager.cc:1397] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.000s
I20250625 13:52:23.820385 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6d0dbbe2894e61b0ea08f98bce61e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea"
I20250625 13:52:23.820479 2775 leader_election.cc:290] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Requested vote from peers 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291), 5c1aa07ab6bc4b0089e46951240a37c9 (127.0.104.1:44801)
W20250625 13:52:23.821244 849 leader_election.cc:343] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c008de91eb444f48da2c4f0b2d7a1ea (127.0.104.3:39291): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.812551 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34371795c8094e73ab4e5dbeedb7aa2f" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d" is_pre_election: true
I20250625 13:52:23.822062 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a6d0dbbe2894e61b0ea08f98bce61e7" candidate_uuid: "affa1c3035b940e88ca143be7d35840d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1aa07ab6bc4b0089e46951240a37c9"
I20250625 13:52:23.822269 764 raft_consensus.cc:3058] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:52:23.822330 1171 raft_consensus.cc:357] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.822580 1171 raft_consensus.cc:383] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:52:23.822657 1171 raft_consensus.cc:738] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Initialized, Role: FOLLOWER
I20250625 13:52:23.822924 1171 consensus_queue.cc:260] T b4be0bc23cde4f0f94bd678096e0c7d8 P affa1c3035b940e88ca143be7d35840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.825309 764 raft_consensus.cc:2466] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P 5c1aa07ab6bc4b0089e46951240a37c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affa1c3035b940e88ca143be7d35840d in term 1.
I20250625 13:52:23.826053 851 leader_election.cc:304] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9, affa1c3035b940e88ca143be7d35840d; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea
I20250625 13:52:23.826520 2775 raft_consensus.cc:2802] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:52:23.826628 2775 raft_consensus.cc:695] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [term 1 LEADER]: Becoming Leader. State: Replica: affa1c3035b940e88ca143be7d35840d, State: Running, Role: LEADER
W20250625 13:52:23.826627 711 leader_election.cc:343] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer affa1c3035b940e88ca143be7d35840d (127.0.104.2:34247): Illegal state: must be running to vote when last-logged opid is not known
I20250625 13:52:23.826766 711 leader_election.cc:304] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c1aa07ab6bc4b0089e46951240a37c9; no voters: 1c008de91eb444f48da2c4f0b2d7a1ea, affa1c3035b940e88ca143be7d35840d
I20250625 13:52:23.826821 2775 consensus_queue.cc:237] T 1a6d0dbbe2894e61b0ea08f98bce61e7 P affa1c3035b940e88ca143be7d35840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c008de91eb444f48da2c4f0b2d7a1ea" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39291 } } peers { permanent_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44801 } } peers { permanent_uuid: "affa1c3035b940e88ca143be7d35840d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34247 } }
I20250625 13:52:23.827399 2903 raft_consensus.cc:2747] T 34371795c8094e73ab4e5dbeedb7aa2f P 5c1aa07ab6bc4b0089e46951240a37c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:52:23.812753 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa16fa387b2a4bb5899b1487642c48d0" candidate_uuid: "5c1aa07ab6bc4b0089e46951240a37c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "affa1c3035b940e88ca143be7d35840d"